[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:01 dist-test-slave-ltqz krb5kdc[21520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:01 dist-test-slave-ltqz krb5kdc[21520](info): set up 2 sockets
Sep 27 07:04:01 dist-test-slave-ltqz krb5kdc[21520](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 07:04:02.756184 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:02 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:02.874261 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43859
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:43859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:02.956763 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:02.957046 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:02.957088 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:02.958636 21533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:02.958714 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:02.958737 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:02.958758 21533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:02.958791 21533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:02.960417 21533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:43859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:02.960722 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:02.960965 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:02.966327 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:02.966404 21533 server_base.cc:1034] running on GCE node
W20260927 07:04:02.966354 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:02.966365 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:02.966737 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:02.967041 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:02.968202 21533 hybrid_clock.cc:648] HybridClock initialized: now 1790492642968191 us; error 45 us; skew 500 ppm
Sep 27 07:04:02 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492642, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:02.970732 21533 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:02.971210 21533 webserver.cc:460] Webserver started at http://127.21.2.126:36001/ using document root <none> and password file <none>
I20260927 07:04:02.971511 21533 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:02.971589 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:02.971725 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:02.972790 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "86f78077fdff40718ec00fdeb814de8e"
format_stamp: "Formatted at 2026-09-27 07:04:02 on dist-test-slave-ltqz"
server_key: "ec17faa8da091ca075f01ef536d22660"
server_key_iv: "a01722e938612e3d1639e37d957fce61"
server_key_version: "encryptionkey@0"
I20260927 07:04:02.973181 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "86f78077fdff40718ec00fdeb814de8e"
format_stamp: "Formatted at 2026-09-27 07:04:02 on dist-test-slave-ltqz"
server_key: "ec17faa8da091ca075f01ef536d22660"
server_key_iv: "a01722e938612e3d1639e37d957fce61"
server_key_version: "encryptionkey@0"
I20260927 07:04:02.974507 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:02.975240 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:02.975438 21533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:02.975549 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "86f78077fdff40718ec00fdeb814de8e"
format_stamp: "Formatted at 2026-09-27 07:04:02 on dist-test-slave-ltqz"
server_key: "ec17faa8da091ca075f01ef536d22660"
server_key_iv: "a01722e938612e3d1639e37d957fce61"
server_key_version: "encryptionkey@0"
I20260927 07:04:02.975634 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:02.995194 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:02.997226 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:02.997398 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:03.001493 21533 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:43859
I20260927 07:04:03.001528 21600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:43859 every 8 connection(s)
I20260927 07:04:03.001876 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:03.002723 21601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:03.005019 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e: Bootstrap starting.
I20260927 07:04:03.005582 21601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:03.005816 21601 log.cc:824] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:03.006461 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e: No bootstrap required, opened a new log
I20260927 07:04:03.008088 21601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } }
I20260927 07:04:03.008224 21601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:03.008247 21601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f78077fdff40718ec00fdeb814de8e, State: Initialized, Role: FOLLOWER
I20260927 07:04:03.008394 21601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } }
I20260927 07:04:03.008489 21601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:03.008518 21601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:03.008548 21601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:03.009116 21601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } }
I20260927 07:04:03.009152 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 21533
I20260927 07:04:03.009202 21601 leader_election.cc:302] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f78077fdff40718ec00fdeb814de8e; no voters:
I20260927 07:04:03.009280 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:03.009394 21601 leader_election.cc:288] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:03.009446 21604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:03.009614 21604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [term 1 LEADER]: Becoming Leader. State: Replica: 86f78077fdff40718ec00fdeb814de8e, State: Running, Role: LEADER
I20260927 07:04:03.009676 21601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:03.009743 21604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } }
I20260927 07:04:03.010114 21513 external_mini_cluster.cc:1428] Setting key c63dd082f023368a5fda34df1cf80c4a
I20260927 07:04:03.010149 21605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f78077fdff40718ec00fdeb814de8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } } }
I20260927 07:04:03.010214 21605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:03.010296 21606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f78077fdff40718ec00fdeb814de8e. Latest consensus state: current_term: 1 leader_uuid: "86f78077fdff40718ec00fdeb814de8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f78077fdff40718ec00fdeb814de8e" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43859 } } }
I20260927 07:04:03.010378 21606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:03.010443 21612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:03.010931 21612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:03.012537 21612 catalog_manager.cc:1347] Generated new cluster ID: 49f30602cbfb4e9992d4b387123de447
I20260927 07:04:03.012588 21612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:03.019625 21612 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:03.020203 21612 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:03.028199 21612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f78077fdff40718ec00fdeb814de8e: Generated new TSK 0
I20260927 07:04:03.028378 21612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:04:03.028708 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.011447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.21.2.126:43859)
0927 07:04:03.011632 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.011641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.012002 (+ 361us) server_negotiation.cc:407] Connection header received
0927 07:04:03.013996 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.014003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.014061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.014189 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.016308 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.017716 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.019317 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.019940 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.020070 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.024321 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.024325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.024326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.024337 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.026297 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.026803 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.026806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.026806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.026907 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.027194 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.027195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.027195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.027481 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.027603 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.028324 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.028474 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:03.054903 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:03.136323 21627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:03.136531 21627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:03.136555 21627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:03.137723 21627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:03.137768 21627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:03.137794 21627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:03.139241 21627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:03.139499 21627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:03.139755 21627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:03.144688 21633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:03.144608 21632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:03.144611 21635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:03.145078 21627 server_base.cc:1034] running on GCE node
I20260927 07:04:03.145264 21627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:03.145502 21627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:03.146692 21627 hybrid_clock.cc:648] HybridClock initialized: now 1790492643146675 us; error 43 us; skew 500 ppm
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:03.149308 21627 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:03.149824 21627 webserver.cc:460] Webserver started at http://127.21.2.65:35569/ using document root <none> and password file <none>
I20260927 07:04:03.150835 21627 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:03.150933 21627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:03.151098 21627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:03.152498 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "7f7402289c80473ea2e116f97cb6b8d4"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "b127a2d0a5d5d404b5d10ff9ce5fa9d2"
server_key_iv: "cbc641883b52ac7d60250e77146575f3"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.152886 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "7f7402289c80473ea2e116f97cb6b8d4"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "b127a2d0a5d5d404b5d10ff9ce5fa9d2"
server_key_iv: "cbc641883b52ac7d60250e77146575f3"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.154326 21627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:04:03.155145 21642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.155323 21627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:03.155516 21627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "7f7402289c80473ea2e116f97cb6b8d4"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "b127a2d0a5d5d404b5d10ff9ce5fa9d2"
server_key_iv: "cbc641883b52ac7d60250e77146575f3"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.155620 21627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:03.167555 21627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:03.169420 21627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:03.169598 21627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:03.169847 21627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:03.170189 21627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:03.170248 21627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.170296 21627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:03.170332 21627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.176829 21627 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:35313
I20260927 07:04:03.176870 21755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:35313 every 8 connection(s)
I20260927 07:04:03.177241 21627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:03.180015 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 21627
I20260927 07:04:03.180140 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:03.180310 21513 external_mini_cluster.cc:1428] Setting key 9b0d88fa8ffffe2e9ffb25d3e47583f8
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:03.188280 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.178011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43859 (local address 127.21.2.65:36489)
0927 07:04:03.178194 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:03.178243 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:03.178761 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:03.179016 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:03.179020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:03.179700 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:03.179952 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.179960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.180987 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.180988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.182309 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.182313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.182889 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.182890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.182993 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.183569 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:03.183578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:03.185030 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:03.186895 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.186900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.186902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.187119 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.187339 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.187339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.187340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.187498 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.187800 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:03.187805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:03.187923 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 07:04:03.187999 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:04:03.188116 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260927 07:04:03.188282 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.178031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:36489 (local address 127.21.2.126:43859)
0927 07:04:03.178090 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.178092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.178331 (+ 239us) server_negotiation.cc:407] Connection header received
0927 07:04:03.178866 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.178868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.178880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.178946 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.180100 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.180890 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.182430 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.182792 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.182878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.185133 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.185135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.185135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.185144 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.186816 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.187196 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.187197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.187198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.187267 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.187577 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.187579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.187579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.187730 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.187808 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.188071 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.188170 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:04:03.188858 21756 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43859
I20260927 07:04:03.188951 21756 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:04:03.189150 21756 heartbeater.cc:511] Master 127.21.2.126:43859 requested a full tablet report, sending...
I20260927 07:04:03.189719 21565 ts_manager.cc:194] Registered new tserver with Master: 7f7402289c80473ea2e116f97cb6b8d4 (127.21.2.65:35313)
I20260927 07:04:03.190376 21565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:36489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:03.204255 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:03.288187 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:03.288556 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:03.288601 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:03.289826 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:03.289902 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:03.289950 21761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:03.291441 21761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:03.291700 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:03.291940 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:03.296911 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:03.296957 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:03.297003 21761 server_base.cc:1034] running on GCE node
W20260927 07:04:03.296926 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:03.297283 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:03.297524 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:03.298684 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790492643298672 us; error 43 us; skew 500 ppm
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:03.301110 21761 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:03.301558 21761 webserver.cc:460] Webserver started at http://127.21.2.66:38015/ using document root <none> and password file <none>
I20260927 07:04:03.301770 21761 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:03.301836 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:03.301977 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:03.302990 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "9c4c00d1873547daa62baba65d82f45c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "4ac7f61695c7357c10dea0e9c900ffb6"
server_key_iv: "46b8784cbeea1dbbb328db22b0f0ccdc"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.303337 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "9c4c00d1873547daa62baba65d82f45c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "4ac7f61695c7357c10dea0e9c900ffb6"
server_key_iv: "46b8784cbeea1dbbb328db22b0f0ccdc"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.304687 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:03.305398 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.305578 21761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:03.305711 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "9c4c00d1873547daa62baba65d82f45c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "4ac7f61695c7357c10dea0e9c900ffb6"
server_key_iv: "46b8784cbeea1dbbb328db22b0f0ccdc"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.305788 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:03.314410 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:03.316256 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:03.316414 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:03.316642 21761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:03.316942 21761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:03.316995 21761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.317037 21761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:03.317063 21761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.323289 21761 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:45171
I20260927 07:04:03.323333 21889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:45171 every 8 connection(s)
I20260927 07:04:03.323673 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:03.329077 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 21761
I20260927 07:04:03.329187 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:03.329362 21513 external_mini_cluster.cc:1428] Setting key 60eddc3cbfed1f563af48ac3e32ad59c
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:03.333408 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.324352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43859 (local address 127.21.2.66:52975)
0927 07:04:03.324525 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:03.324567 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:03.325050 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:03.325252 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:03.325255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:03.325838 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:03.326039 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.326044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.326823 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.326824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.327656 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.327658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.328016 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.328016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.328126 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.328620 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:03.328627 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:03.330230 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:03.331954 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.331961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.331964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.332282 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.332483 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.332484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.332484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.332633 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.332927 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:03.332931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:03.333048 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 07:04:03.333129 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:04:03.333247 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260927 07:04:03.333406 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.324374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:52975 (local address 127.21.2.126:43859)
0927 07:04:03.324445 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.324447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.324653 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:04:03.325130 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.325131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.325140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.325199 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.326151 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.326645 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.327740 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.327935 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.327999 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.330323 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.330324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.330325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.330332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.331867 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.332367 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.332368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.332368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.332417 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.332708 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.332709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.332709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.332858 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.332933 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.333175 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.333263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:04:03.333945 21890 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43859
I20260927 07:04:03.334056 21890 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:03.334223 21890 heartbeater.cc:511] Master 127.21.2.126:43859 requested a full tablet report, sending...
I20260927 07:04:03.334636 21565 ts_manager.cc:194] Registered new tserver with Master: 9c4c00d1873547daa62baba65d82f45c (127.21.2.66:45171)
I20260927 07:04:03.335183 21565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:52975
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:03.354357 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:03.433944 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:03.434182 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:03.434221 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:03.435340 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:03.435459 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:03.435508 21895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:03.436928 21895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:03.437184 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:03.437415 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:03.442476 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:03.442512 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:03.442471 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:03.442763 21895 server_base.cc:1034] running on GCE node
I20260927 07:04:03.443001 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:03.443284 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:03.444439 21895 hybrid_clock.cc:648] HybridClock initialized: now 1790492643444448 us; error 52 us; skew 500 ppm
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:03.447121 21895 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:03.447583 21895 webserver.cc:460] Webserver started at http://127.21.2.67:35375/ using document root <none> and password file <none>
I20260927 07:04:03.447770 21895 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:03.447810 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:03.447899 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:03.448985 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "83701571eb3e49d393209a8601849e3c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "0453261880fcdc79e428c66900a458c3"
server_key_iv: "351f5a5f083b4cc02233bb4898a0c3ed"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.449306 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "83701571eb3e49d393209a8601849e3c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "0453261880fcdc79e428c66900a458c3"
server_key_iv: "351f5a5f083b4cc02233bb4898a0c3ed"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.451149 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:04:03.452221 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.452483 21895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:03.452615 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "83701571eb3e49d393209a8601849e3c"
format_stamp: "Formatted at 2026-09-27 07:04:03 on dist-test-slave-ltqz"
server_key: "0453261880fcdc79e428c66900a458c3"
server_key_iv: "351f5a5f083b4cc02233bb4898a0c3ed"
server_key_version: "encryptionkey@0"
I20260927 07:04:03.452705 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:03.470021 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:03.472165 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:03.472333 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:03.472584 21895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:03.472896 21895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:03.472951 21895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.472993 21895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:03.473022 21895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:03.479238 21895 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:42153
I20260927 07:04:03.479276 22023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:42153 every 8 connection(s)
I20260927 07:04:03.479647 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:03.488691 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.480383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:58737 (local address 127.21.2.126:43859)
0927 07:04:03.480430 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.480432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.480645 (+ 213us) server_negotiation.cc:407] Connection header received
0927 07:04:03.481194 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.481195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.481208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.481260 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.482225 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.482729 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.483826 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.484037 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.484124 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.486248 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.486250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.486250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.486257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.487505 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.487893 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.487894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.487894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.487945 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.488154 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.488154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.488154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.488255 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.488304 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.488565 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.488626 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:03.488869 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.480354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43859 (local address 127.21.2.67:58737)
0927 07:04:03.480520 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:03.480555 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:03.481073 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:03.481329 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:03.481333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:03.481915 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:03.482111 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.482114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.482824 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.482825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.483720 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:03.483723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.484134 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.484136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:03.484259 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.484740 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:03.484748 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:03.486160 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:03.487594 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.487601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.487603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.487824 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.488004 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:03.488005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:03.488005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:03.488100 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:03.488310 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:03.488321 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:03.488440 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 07:04:03.488536 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 07:04:03.488657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260927 07:04:03.489408 22024 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43859
I20260927 07:04:03.489421 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 21895
I20260927 07:04:03.489524 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:03.489528 22024 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:03.489693 21513 external_mini_cluster.cc:1428] Setting key 2e790c32aad6f653ce02ec432a8e72e9
I20260927 07:04:03.489739 22024 heartbeater.cc:511] Master 127.21.2.126:43859 requested a full tablet report, sending...
I20260927 07:04:03.490173 21565 ts_manager.cc:194] Registered new tserver with Master: 83701571eb3e49d393209a8601849e3c (127.21.2.67:42153)
I20260927 07:04:03.490685 21565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:58737
I20260927 07:04:03.490862 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:03.514556 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.506149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.21.2.65:35313)
0927 07:04:03.506299 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.506304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.506325 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:04:03.506410 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.506412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.506425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.506500 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.507081 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.507869 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.508673 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.509102 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.509166 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.512001 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.512009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.512009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.512017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.513550 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.513894 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.513897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.513897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.513953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.514115 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.514116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.514116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.514241 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.514349 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.514414 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.514474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260927 07:04:03.514976 21670 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[21520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:03.537205 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:03.530416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.21.2.65:35313)
0927 07:04:03.530514 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:03.530516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:03.530583 (+ 67us) server_negotiation.cc:407] Connection header received
0927 07:04:03.530663 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:03.530665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:03.530677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:03.530744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:03.531275 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.531831 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.532583 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:03.532793 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:03.532879 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:03.535094 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:03.535095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:03.535096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:03.535103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:03.536330 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.536621 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.536622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.536622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.536671 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:03.536828 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:03.536829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:03.536829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:03.536915 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:03.537004 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:03.537079 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 07:04:03.537140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:04:03.537740 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 21627
I20260927 07:04:03.542191 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 21761
I20260927 07:04:03.546424 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 21895
I20260927 07:04:03.552058 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 21533
2026-09-27T07:04:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1855 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[22038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[22038](info): set up 2 sockets
Sep 27 07:04:03 dist-test-slave-ltqz krb5kdc[22038](info): commencing operation
krb5kdc: starting...
W20260927 07:04:04.590781 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:04 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:04.707676 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35475
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:04.790091 22051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:04.790347 22051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:04.790386 22051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:04.791572 22051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:04.791644 22051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:04.791671 22051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:04.791703 22051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:04.791728 22051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:04.793247 22051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:35475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:04.793560 22051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:04.793795 22051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:04.798688 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:04.798748 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:04.798866 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:04.799444 22051 server_base.cc:1034] running on GCE node
I20260927 07:04:04.799715 22051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:04.800024 22051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:04.801198 22051 hybrid_clock.cc:648] HybridClock initialized: now 1790492644801187 us; error 44 us; skew 500 ppm
Sep 27 07:04:04 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:04.803741 22051 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:04.804220 22051 webserver.cc:460] Webserver started at http://127.21.2.126:41325/ using document root <none> and password file <none>
I20260927 07:04:04.804463 22051 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:04.804533 22051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:04.804684 22051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:04.805749 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "4efd9b15a36d4c70a8eec022bdedaf0a"
format_stamp: "Formatted at 2026-09-27 07:04:04 on dist-test-slave-ltqz"
server_key: "83722a8ee4991bc18efa26a8148fd656"
server_key_iv: "f3fc1de4ebed6b6073ea56cb9ff169dc"
server_key_version: "encryptionkey@0"
I20260927 07:04:04.806124 22051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "4efd9b15a36d4c70a8eec022bdedaf0a"
format_stamp: "Formatted at 2026-09-27 07:04:04 on dist-test-slave-ltqz"
server_key: "83722a8ee4991bc18efa26a8148fd656"
server_key_iv: "f3fc1de4ebed6b6073ea56cb9ff169dc"
server_key_version: "encryptionkey@0"
I20260927 07:04:04.807520 22051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:04:04.808266 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:04.808437 22051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:04.808552 22051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "4efd9b15a36d4c70a8eec022bdedaf0a"
format_stamp: "Formatted at 2026-09-27 07:04:04 on dist-test-slave-ltqz"
server_key: "83722a8ee4991bc18efa26a8148fd656"
server_key_iv: "f3fc1de4ebed6b6073ea56cb9ff169dc"
server_key_version: "encryptionkey@0"
I20260927 07:04:04.808626 22051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:04.842005 22051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:04.843969 22051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:04.844141 22051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:04.848786 22051 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:35475
I20260927 07:04:04.848838 22118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:35475 every 8 connection(s)
I20260927 07:04:04.849205 22051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:04.850066 22119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:04.852659 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a: Bootstrap starting.
I20260927 07:04:04.852834 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22051
I20260927 07:04:04.852923 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:04.853083 21513 external_mini_cluster.cc:1428] Setting key a95800a4ceb331eba4d00c823ea5fc7c
I20260927 07:04:04.853402 22119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:04.853762 22119 log.cc:824] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:04.854661 22119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a: No bootstrap required, opened a new log
I20260927 07:04:04.857131 22119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } }
I20260927 07:04:04.857314 22119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:04.857362 22119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efd9b15a36d4c70a8eec022bdedaf0a, State: Initialized, Role: FOLLOWER
I20260927 07:04:04.857504 22119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } }
I20260927 07:04:04.857586 22119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:04.857621 22119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:04.857673 22119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:04.858315 22119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } }
I20260927 07:04:04.858423 22119 leader_election.cc:302] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efd9b15a36d4c70a8eec022bdedaf0a; no voters:
I20260927 07:04:04.858582 22119 leader_election.cc:288] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:04.858664 22124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:04.858819 22124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [term 1 LEADER]: Becoming Leader. State: Replica: 4efd9b15a36d4c70a8eec022bdedaf0a, State: Running, Role: LEADER
I20260927 07:04:04.858832 22119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:04.858903 22124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } }
I20260927 07:04:04.859323 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } } }
I20260927 07:04:04.859422 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:04.859835 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efd9b15a36d4c70a8eec022bdedaf0a. Latest consensus state: current_term: 1 leader_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efd9b15a36d4c70a8eec022bdedaf0a" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35475 } } }
I20260927 07:04:04.859915 22131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:04.859978 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:04.860451 22131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:04:04 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:04.862156 22131 catalog_manager.cc:1347] Generated new cluster ID: b46c4c6b1f114bfe9ebb68bcdf081bc3
I20260927 07:04:04.862207 22131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:04.865744 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:04.853629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.21.2.126:35475)
0927 07:04:04.853870 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:04.853880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:04.854148 (+ 268us) server_negotiation.cc:407] Connection header received
0927 07:04:04.854818 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:04.854826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:04.854887 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:04.855042 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:04.855689 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:04.857148 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:04.858107 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:04.858759 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:04.860907 (+ 2148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:04.861974 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:04.861978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:04.861978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:04.861990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:04.864079 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:04.864473 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:04.864476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:04.864476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:04.864592 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:04.864796 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:04.864797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:04.864797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:04.865114 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:04.865268 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:04.865374 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:04:04.865516 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
I20260927 07:04:04.876368 22131 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:04.877120 22131 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:04.894543 22131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4efd9b15a36d4c70a8eec022bdedaf0a: Generated new TSK 0
I20260927 07:04:04.894774 22131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:04.926589 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:05.008414 22145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:05.008630 22145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:05.008671 22145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:05.009814 22145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:05.009876 22145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:05.009922 22145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:05.011595 22145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:05.011845 22145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:05.012084 22145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:05.017001 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:05.017014 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:05.017078 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:05.017263 22145 server_base.cc:1034] running on GCE node
I20260927 07:04:05.017465 22145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:05.017664 22145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:05.018810 22145 hybrid_clock.cc:648] HybridClock initialized: now 1790492645018794 us; error 30 us; skew 500 ppm
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:05.021488 22145 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:05.021912 22145 webserver.cc:460] Webserver started at http://127.21.2.65:39537/ using document root <none> and password file <none>
I20260927 07:04:05.022151 22145 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:05.022220 22145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:05.022368 22145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:05.023458 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "193764d46f8044fbb3f83138ee0c6d0d"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "87d743d5155e3da1f232057daa35e469"
server_key_iv: "eeb87fa86cd070d2b424329f77eaf551"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.023818 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "193764d46f8044fbb3f83138ee0c6d0d"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "87d743d5155e3da1f232057daa35e469"
server_key_iv: "eeb87fa86cd070d2b424329f77eaf551"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.025146 22145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:05.025892 22160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.026053 22145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:05.026154 22145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "193764d46f8044fbb3f83138ee0c6d0d"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "87d743d5155e3da1f232057daa35e469"
server_key_iv: "eeb87fa86cd070d2b424329f77eaf551"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.026226 22145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:05.039855 22145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:05.041658 22145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:05.041810 22145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:05.042033 22145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:05.042337 22145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:05.042390 22145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.042430 22145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:05.042466 22145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.048199 22145 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37759
I20260927 07:04:05.048272 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37759 every 8 connection(s)
I20260927 07:04:05.048599 22145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:05.051868 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22145
I20260927 07:04:05.051970 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:05.052124 21513 external_mini_cluster.cc:1428] Setting key adfd69ff3f74178bd8182f57801fce43
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:05.059249 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.049363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:52937 (local address 127.21.2.126:35475)
0927 07:04:05.049405 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.049407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.049631 (+ 224us) server_negotiation.cc:407] Connection header received
0927 07:04:05.050635 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.050636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.050648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.050707 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.051321 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.052161 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.053250 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.053596 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.053671 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.056218 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.056220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.056221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.056230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.057714 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.058217 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.058218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.058218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.058266 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.058555 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.058555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.058556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.058674 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.058727 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.059088 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.059180 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:04:05.059494 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.049341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35475 (local address 127.21.2.65:52937)
0927 07:04:05.049508 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.049549 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.050470 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.050762 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.050765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.051024 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:05.051215 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.051218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.052291 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.052293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.053153 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.053156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.053704 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.053705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.053818 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.054419 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:05.054433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:05.056108 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:05.057798 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.057806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.057809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.058142 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.058331 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.058331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.058332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.058482 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.058743 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:05.058749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:05.058928 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.059046 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.059226 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260927 07:04:05.060053 22274 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35475
I20260927 07:04:05.060168 22274 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:05.060343 22274 heartbeater.cc:511] Master 127.21.2.126:35475 requested a full tablet report, sending...
I20260927 07:04:05.060931 22083 ts_manager.cc:194] Registered new tserver with Master: 193764d46f8044fbb3f83138ee0c6d0d (127.21.2.65:37759)
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:04:05.061683 22083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:52937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:05.077685 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:05.157110 22279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:05.157327 22279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:05.157351 22279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:05.158511 22279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:05.158556 22279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:05.158583 22279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:05.160254 22279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:05.160504 22279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:05.160737 22279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:05.165676 22284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:05.165683 22285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:05.165856 22279 server_base.cc:1034] running on GCE node
W20260927 07:04:05.165712 22287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:05.166162 22279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:05.166422 22279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:05.167582 22279 hybrid_clock.cc:648] HybridClock initialized: now 1790492645167569 us; error 46 us; skew 500 ppm
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:05.170053 22279 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:05.170517 22279 webserver.cc:460] Webserver started at http://127.21.2.66:42993/ using document root <none> and password file <none>
I20260927 07:04:05.170745 22279 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:05.170814 22279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:05.170964 22279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:05.172092 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "2accdf407aa849d38c423b91615a8058"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "3abfe86efe4358f46f8f44ab39e318e8"
server_key_iv: "5fede56a8d0dc9ed1980b9892f3f1483"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.172459 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "2accdf407aa849d38c423b91615a8058"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "3abfe86efe4358f46f8f44ab39e318e8"
server_key_iv: "5fede56a8d0dc9ed1980b9892f3f1483"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.173746 22279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:05.174463 22294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.174636 22279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:05.174736 22279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "2accdf407aa849d38c423b91615a8058"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "3abfe86efe4358f46f8f44ab39e318e8"
server_key_iv: "5fede56a8d0dc9ed1980b9892f3f1483"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.174815 22279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:05.187937 22279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:05.190223 22279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:05.190375 22279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:05.190594 22279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:05.190898 22279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:05.190968 22279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.191002 22279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:05.191023 22279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.197255 22279 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:42411
I20260927 07:04:05.197300 22407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:42411 every 8 connection(s)
I20260927 07:04:05.197606 22279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:05.202675 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22279
I20260927 07:04:05.202775 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:05.202939 21513 external_mini_cluster.cc:1428] Setting key 1095c244d46972de45a56e8113c932c2
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:05.207866 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.198321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47487 (local address 127.21.2.126:35475)
0927 07:04:05.198391 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.198393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.198598 (+ 205us) server_negotiation.cc:407] Connection header received
0927 07:04:05.199633 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.199635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.199647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.199710 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.200336 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.200834 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.201836 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.202036 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.202132 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.204672 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.204674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.204674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.204681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.206135 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.206649 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.206650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.206651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.206709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.207048 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.207049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.207050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.207182 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.207218 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.207693 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.207775 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:04:05.208051 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.198284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35475 (local address 127.21.2.66:47487)
0927 07:04:05.198467 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.198512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.199521 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.199778 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.199782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.200045 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:05.200234 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.200237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.200924 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.200925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.201731 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.201733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.202131 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.202132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.202265 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.202908 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:05.202916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:05.204554 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:05.206239 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.206247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.206249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.206564 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.206783 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.206784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.206785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.206948 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.207290 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:05.207295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:05.207509 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.207654 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.207815 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260927 07:04:05.208561 22408 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35475
I20260927 07:04:05.208662 22408 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:05.208868 22408 heartbeater.cc:511] Master 127.21.2.126:35475 requested a full tablet report, sending...
I20260927 07:04:05.209297 22083 ts_manager.cc:194] Registered new tserver with Master: 2accdf407aa849d38c423b91615a8058 (127.21.2.66:42411)
I20260927 07:04:05.209848 22083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:47487
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:05.230275 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:05.310067 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:05.310315 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:05.310359 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:05.311682 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:05.311746 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:05.311794 22413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:05.313632 22413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:05.313884 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:05.314122 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:05.319545 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:05.319540 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:05.319656 22413 server_base.cc:1034] running on GCE node
W20260927 07:04:05.319540 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:05.319948 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:05.320204 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:05.321341 22413 hybrid_clock.cc:648] HybridClock initialized: now 1790492645321320 us; error 36 us; skew 500 ppm
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:05.323809 22413 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:05.324225 22413 webserver.cc:460] Webserver started at http://127.21.2.67:46361/ using document root <none> and password file <none>
I20260927 07:04:05.324512 22413 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:05.324580 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:05.324719 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:05.325732 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "2fab5f6c1d814340bc635e0a048340ad"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "dba6655d0999961809d04bca957ff153"
server_key_iv: "a0cf23dd39dbbe1e9a5b08cd7b670112"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.326094 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "2fab5f6c1d814340bc635e0a048340ad"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "dba6655d0999961809d04bca957ff153"
server_key_iv: "a0cf23dd39dbbe1e9a5b08cd7b670112"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.327430 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:05.328162 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.328346 22413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:05.328442 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "2fab5f6c1d814340bc635e0a048340ad"
format_stamp: "Formatted at 2026-09-27 07:04:05 on dist-test-slave-ltqz"
server_key: "dba6655d0999961809d04bca957ff153"
server_key_iv: "a0cf23dd39dbbe1e9a5b08cd7b670112"
server_key_version: "encryptionkey@0"
I20260927 07:04:05.328518 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:05.345105 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:05.346837 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:05.346953 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:05.347142 22413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:05.347515 22413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:05.347551 22413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.347573 22413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:05.347620 22413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:05.353385 22413 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:38847
I20260927 07:04:05.353427 22541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:38847 every 8 connection(s)
I20260927 07:04:05.353731 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:05.354840 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22413
I20260927 07:04:05.354914 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:05.355088 21513 external_mini_cluster.cc:1428] Setting key f18c4f7723b3bc3223fa61e0bf55db79
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:05.363584 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.354440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:43493 (local address 127.21.2.126:35475)
0927 07:04:05.354486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.354488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.354731 (+ 243us) server_negotiation.cc:407] Connection header received
0927 07:04:05.356008 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.356009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.356021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.356078 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.356721 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.357228 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.358251 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.358451 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.358544 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.360901 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.360903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.360904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.360918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.362193 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.362602 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.362604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.362604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.362667 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.362916 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.362917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.362917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.363035 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.363113 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.363402 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.363486 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:05.363794 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.354409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35475 (local address 127.21.2.67:43493)
0927 07:04:05.354596 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.354641 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.355904 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.356151 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.356154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.356426 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:05.356615 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.356619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.357333 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.357334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.358157 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.358159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.358543 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.358544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.358686 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.359281 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:05.359290 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:05.360796 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:05.362295 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.362302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.362304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.362523 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.362740 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:05.362741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:05.362741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:05.362850 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:05.363103 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:05.363109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:05.363255 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.363365 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.363547 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260927 07:04:05.364302 22542 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35475
I20260927 07:04:05.364383 22542 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:05.364593 22542 heartbeater.cc:511] Master 127.21.2.126:35475 requested a full tablet report, sending...
I20260927 07:04:05.365003 22083 ts_manager.cc:194] Registered new tserver with Master: 2fab5f6c1d814340bc635e0a048340ad (127.21.2.67:38847)
I20260927 07:04:05.365491 22083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:43493
I20260927 07:04:05.366880 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:04:05.374514 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.368447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.21.2.126:35475)
0927 07:04:05.368551 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.368554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.368664 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:04:05.368745 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.368746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.368755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.368826 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.369398 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.369914 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.370678 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.370871 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.370945 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.372204 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.372206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.372206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.372213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.373354 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.373701 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.373702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.373703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.373758 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.373960 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.373961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.373962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.374113 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.374221 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.374333 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.374411 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:04:05.376801 22083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:05.378417 22083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:05.384730 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.380682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37759 (local address 127.0.0.1:43828)
0927 07:04:05.380781 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.380789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.380826 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.381456 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.381457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.381460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:05.381506 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.381508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382702 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.382703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.383647 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.383650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.384528 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.384528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.384580 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.384590 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.384643 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.384679 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":36,"threads_started":1}
I20260927 07:04:05.384781 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.381079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.21.2.65:37759)
0927 07:04:05.381288 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.381292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.381301 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:05.381336 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.381338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.381352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.381587 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:05.381657 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382604 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.383732 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.384444 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.384524 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.384586 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.384667 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.384703 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260927 07:04:05.385155 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.380458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:38847 (local address 127.0.0.1:50744)
0927 07:04:05.380578 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.380614 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.380678 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.381327 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.381328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.381330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:05.381378 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.381380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382292 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.382293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.383721 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.383723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.384939 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.384940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.384994 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.384997 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.385039 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.385074 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":67,"thread_start_us":71,"threads_started":1}
I20260927 07:04:05.385401 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.380790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.21.2.67:38847)
0927 07:04:05.381086 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.381091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.381101 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:05.381138 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.381140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.381157 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.381227 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:05.381455 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382185 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.383789 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.384448 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.384524 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.384582 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.385279 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.385326 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":81,"threads_started":1}
I20260927 07:04:05.385435 22208 tablet_service.cc:1467] Processing CreateTablet for tablet 641e2811b6224c3280424e70565b855f (DEFAULT_TABLE table=test-table [id=fef29e8ff1aa4bb3a25e797f200854ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:05.385789 22208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641e2811b6224c3280424e70565b855f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:05.386023 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.380467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:42411 (local address 127.0.0.1:45510)
0927 07:04:05.380752 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.380763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.380823 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.381017 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.381020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.381023 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:05.381086 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.381088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382393 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.382393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.383721 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.383723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.385797 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.385798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.385860 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.385863 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.385919 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.385955 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":87,"thread_start_us":58,"threads_started":1}
I20260927 07:04:05.386039 22476 tablet_service.cc:1467] Processing CreateTablet for tablet 641e2811b6224c3280424e70565b855f (DEFAULT_TABLE table=test-table [id=fef29e8ff1aa4bb3a25e797f200854ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:05.386297 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.380572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.21.2.66:42411)
0927 07:04:05.380750 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.380755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.380811 (+ 56us) server_negotiation.cc:407] Connection header received
0927 07:04:05.380890 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.380892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.380907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.380977 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:05.381174 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.382297 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.384620 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.385413 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.385478 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.385532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.386176 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.386224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260927 07:04:05.386358 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641e2811b6224c3280424e70565b855f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:05.386873 22342 tablet_service.cc:1467] Processing CreateTablet for tablet 641e2811b6224c3280424e70565b855f (DEFAULT_TABLE table=test-table [id=fef29e8ff1aa4bb3a25e797f200854ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:05.387202 22342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 641e2811b6224c3280424e70565b855f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:05.389207 22557 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Bootstrap starting.
I20260927 07:04:05.389806 22557 tablet_bootstrap.cc:654] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:05.390251 22558 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Bootstrap starting.
I20260927 07:04:05.390352 22557 log.cc:824] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:05.390457 22556 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Bootstrap starting.
I20260927 07:04:05.390990 22558 tablet_bootstrap.cc:654] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:05.391072 22556 tablet_bootstrap.cc:654] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:05.391075 22557 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: No bootstrap required, opened a new log
I20260927 07:04:05.391136 22557 ts_tablet_manager.cc:1397] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:05.391675 22558 log.cc:824] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:05.391675 22556 log.cc:824] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:05.392457 22556 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: No bootstrap required, opened a new log
I20260927 07:04:05.392513 22556 ts_tablet_manager.cc:1397] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:05.392952 22557 raft_consensus.cc:348] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.393122 22557 raft_consensus.cc:374] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:05.393168 22557 raft_consensus.cc:729] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193764d46f8044fbb3f83138ee0c6d0d, State: Initialized, Role: FOLLOWER
I20260927 07:04:05.393285 22557 consensus_queue.cc:260] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.393569 22274 heartbeater.cc:503] Master 127.21.2.126:35475 was elected leader, sending a full tablet report...
I20260927 07:04:05.393555 22557 ts_tablet_manager.cc:1428] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:04:05.394246 22556 raft_consensus.cc:348] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.394434 22556 raft_consensus.cc:374] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:05.394483 22556 raft_consensus.cc:729] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fab5f6c1d814340bc635e0a048340ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:05.394603 22556 consensus_queue.cc:260] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.394927 22556 ts_tablet_manager.cc:1428] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:05.394942 22542 heartbeater.cc:503] Master 127.21.2.126:35475 was elected leader, sending a full tablet report...
I20260927 07:04:05.395401 22558 tablet_bootstrap.cc:492] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: No bootstrap required, opened a new log
I20260927 07:04:05.395476 22558 ts_tablet_manager.cc:1397] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 07:04:05.396965 22558 raft_consensus.cc:348] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.397097 22558 raft_consensus.cc:374] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:05.397120 22558 raft_consensus.cc:729] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2accdf407aa849d38c423b91615a8058, State: Initialized, Role: FOLLOWER
I20260927 07:04:05.397226 22558 consensus_queue.cc:260] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.397495 22558 ts_tablet_manager.cc:1428] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 07:04:05.397516 22408 heartbeater.cc:503] Master 127.21.2.126:35475 was elected leader, sending a full tablet report...
W20260927 07:04:05.448179 22409 tablet.cc:2369] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:04:05.549372 22275 tablet.cc:2369] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:05.566413 22563 raft_consensus.cc:482] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:05.566532 22563 raft_consensus.cc:504] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.566883 22563 leader_election.cc:288] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2accdf407aa849d38c423b91615a8058 (127.21.2.66:42411), 193764d46f8044fbb3f83138ee0c6d0d (127.21.2.65:37759)
I20260927 07:04:05.570014 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.567307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:50729 (local address 127.21.2.65:37759)
0927 07:04:05.567357 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.567378 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.567448 (+ 70us) server_negotiation.cc:407] Connection header received
0927 07:04:05.567497 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.567498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.567511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.567585 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:05.567739 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.568203 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569227 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.569571 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569757 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.569768 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.569887 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.569937 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:04:05.570048 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.567045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:42411 (local address 127.21.2.67:37929)
0927 07:04:05.567105 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.567119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.567164 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.567399 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.567400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.567402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:05.567501 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.567502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.568202 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.568203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.569088 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.569090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.569749 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.569831 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.569836 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.569917 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.569953 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:04:05.570050 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.567133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:37929 (local address 127.21.2.66:42411)
0927 07:04:05.567214 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.567216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.567227 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:04:05.567257 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.567258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.567271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.567355 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:05.567588 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.568122 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569183 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.569569 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569655 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.569666 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.569939 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.569984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:04:05.570060 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.567185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37759 (local address 127.21.2.67:50729)
0927 07:04:05.567353 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:05.567384 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:05.567444 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:05.567601 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:05.567601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:05.567603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:05.567671 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.567672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.568283 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.568284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.569102 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:05.569104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.569756 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.569757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:05.569814 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.569818 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:05.569861 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 07:04:05.569913 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260927 07:04:05.570377 22228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641e2811b6224c3280424e70565b855f" candidate_uuid: "2fab5f6c1d814340bc635e0a048340ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193764d46f8044fbb3f83138ee0c6d0d" is_pre_election: true
I20260927 07:04:05.570456 22362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641e2811b6224c3280424e70565b855f" candidate_uuid: "2fab5f6c1d814340bc635e0a048340ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2accdf407aa849d38c423b91615a8058" is_pre_election: true
I20260927 07:04:05.570554 22228 raft_consensus.cc:2454] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fab5f6c1d814340bc635e0a048340ad in term 0.
I20260927 07:04:05.570593 22362 raft_consensus.cc:2454] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fab5f6c1d814340bc635e0a048340ad in term 0.
I20260927 07:04:05.570791 22432 leader_election.cc:302] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2accdf407aa849d38c423b91615a8058, 2fab5f6c1d814340bc635e0a048340ad; no voters:
I20260927 07:04:05.570942 22563 raft_consensus.cc:2781] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:05.571051 22563 raft_consensus.cc:482] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:05.571085 22563 raft_consensus.cc:3037] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:05.571870 22563 raft_consensus.cc:504] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.572016 22563 leader_election.cc:288] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [CANDIDATE]: Term 1 election: Requested vote from peers 2accdf407aa849d38c423b91615a8058 (127.21.2.66:42411), 193764d46f8044fbb3f83138ee0c6d0d (127.21.2.65:37759)
I20260927 07:04:05.572233 22362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641e2811b6224c3280424e70565b855f" candidate_uuid: "2fab5f6c1d814340bc635e0a048340ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2accdf407aa849d38c423b91615a8058"
I20260927 07:04:05.572316 22362 raft_consensus.cc:3037] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:05.572321 22228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "641e2811b6224c3280424e70565b855f" candidate_uuid: "2fab5f6c1d814340bc635e0a048340ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193764d46f8044fbb3f83138ee0c6d0d"
I20260927 07:04:05.572458 22228 raft_consensus.cc:3037] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:05.573079 22362 raft_consensus.cc:2454] T 641e2811b6224c3280424e70565b855f P 2accdf407aa849d38c423b91615a8058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fab5f6c1d814340bc635e0a048340ad in term 1.
I20260927 07:04:05.573230 22228 raft_consensus.cc:2454] T 641e2811b6224c3280424e70565b855f P 193764d46f8044fbb3f83138ee0c6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fab5f6c1d814340bc635e0a048340ad in term 1.
I20260927 07:04:05.573346 22432 leader_election.cc:302] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2accdf407aa849d38c423b91615a8058, 2fab5f6c1d814340bc635e0a048340ad; no voters:
I20260927 07:04:05.573452 22563 raft_consensus.cc:2781] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:05.573657 22563 raft_consensus.cc:686] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [term 1 LEADER]: Becoming Leader. State: Replica: 2fab5f6c1d814340bc635e0a048340ad, State: Running, Role: LEADER
I20260927 07:04:05.573771 22563 consensus_queue.cc:237] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } }
I20260927 07:04:05.574643 22083 catalog_manager.cc:5495] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fab5f6c1d814340bc635e0a048340ad (127.21.2.67). New cstate: current_term: 1 leader_uuid: "2fab5f6c1d814340bc635e0a048340ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fab5f6c1d814340bc635e0a048340ad" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2accdf407aa849d38c423b91615a8058" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "193764d46f8044fbb3f83138ee0c6d0d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37759 } health_report { overall_health: UNKNOWN } } }
W20260927 07:04:05.604262 22543 tablet.cc:2369] T 641e2811b6224c3280424e70565b855f P 2fab5f6c1d814340bc635e0a048340ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:05.618381 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.610947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.21.2.65:37759)
0927 07:04:05.611019 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.611021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.611168 (+ 147us) server_negotiation.cc:407] Connection header received
0927 07:04:05.611209 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.611210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.611220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.611273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.611888 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.612432 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.613114 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.613299 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.613350 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.615715 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.615725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.615726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.615733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.617130 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.617537 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.617541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.617541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.617616 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.617820 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.617821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.617821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.617963 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.618122 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.618193 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.618273 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:05.640750 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.633662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.21.2.65:37759)
0927 07:04:05.633750 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.633752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.633781 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:04:05.633852 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.633853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.633862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.633911 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.634427 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.634884 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.635827 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.636005 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.636057 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.638325 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.638327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.638327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.638333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.639735 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.640048 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.640049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.640049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.640096 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.640262 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.640262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.640263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.640345 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.640414 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.640566 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.640624 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:05.662456 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:05.655319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.21.2.65:37759)
0927 07:04:05.655446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:05.655448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:05.655472 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:04:05.655542 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:05.655543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:05.655552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:05.655599 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:05.656098 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.656548 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.657233 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:05.657402 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:05.657450 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:05.659913 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:05.659915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:05.659915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:05.659923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:05.661339 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.661703 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.661704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.661705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.661762 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:05.661959 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:05.661960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:05.661961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:05.662058 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:05.662166 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:05.662274 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 07:04:05.662348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 07:04:05.662894 22188 tablet_service.cc:2908] Rejecting scan request for tablet 641e2811b6224c3280424e70565b855f: Uninitialized: clean time has not yet been initialized
I20260927 07:04:05.663689 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22145
I20260927 07:04:05.668197 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22279
I20260927 07:04:05.672883 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22413
I20260927 07:04:05.677167 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22051
2026-09-27T07:04:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2124 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22580](info): set up 2 sockets
Sep 27 07:04:05 dist-test-slave-ltqz krb5kdc[22580](info): commencing operation
krb5kdc: starting...
W20260927 07:04:06.706928 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:06 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:06.823524 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34793
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:06.903577 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:06.903805 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:06.903827 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:06.904933 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:06.904971 22593 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:04:06.904984 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:06.904997 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:06.905009 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:06.905021 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:06.906440 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34793
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:06.906651 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:06.906827 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:06.911686 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:06.911705 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:06.911705 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:06.911904 22593 server_base.cc:1034] running on GCE node
I20260927 07:04:06.912273 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:06.912616 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:06.913758 22593 hybrid_clock.cc:648] HybridClock initialized: now 1790492646913739 us; error 41 us; skew 500 ppm
Sep 27 07:04:06 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492646, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:06.922032 22593 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:06.922510 22593 webserver.cc:460] Webserver started at http://127.21.2.126:38681/ using document root <none> and password file <none>
I20260927 07:04:06.922744 22593 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:06.922789 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:06.922950 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:06.924080 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "2ae2722b82784c95ac13bbb3db7f5c40"
format_stamp: "Formatted at 2026-09-27 07:04:06 on dist-test-slave-ltqz"
server_key: "eb14a06e5ea94f084949692c97ab1930"
server_key_iv: "62daaecd113219a7a80d465d1aa42e6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:06.924451 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "2ae2722b82784c95ac13bbb3db7f5c40"
format_stamp: "Formatted at 2026-09-27 07:04:06 on dist-test-slave-ltqz"
server_key: "eb14a06e5ea94f084949692c97ab1930"
server_key_iv: "62daaecd113219a7a80d465d1aa42e6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:06.925814 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:06.926587 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:06.926829 22593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:06.926939 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "2ae2722b82784c95ac13bbb3db7f5c40"
format_stamp: "Formatted at 2026-09-27 07:04:06 on dist-test-slave-ltqz"
server_key: "eb14a06e5ea94f084949692c97ab1930"
server_key_iv: "62daaecd113219a7a80d465d1aa42e6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:06.927014 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:06.937562 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:06.939893 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:06.940059 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:06.944679 22593 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34793
I20260927 07:04:06.944741 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34793 every 8 connection(s)
I20260927 07:04:06.945047 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:06.945633 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:06.947932 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Bootstrap starting.
I20260927 07:04:06.948203 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22593
I20260927 07:04:06.948284 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:06.948443 21513 external_mini_cluster.cc:1428] Setting key c13e8a447483652263634306bd81331a
I20260927 07:04:06.948680 22671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:06.949012 22671 log.cc:824] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:06.949754 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: No bootstrap required, opened a new log
I20260927 07:04:06.951591 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } }
I20260927 07:04:06.951769 22671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:06.951803 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae2722b82784c95ac13bbb3db7f5c40, State: Initialized, Role: FOLLOWER
I20260927 07:04:06.951920 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } }
I20260927 07:04:06.951992 22671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:06.952020 22671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:06.952055 22671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:06.952700 22671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } }
I20260927 07:04:06.952790 22671 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae2722b82784c95ac13bbb3db7f5c40; no voters:
I20260927 07:04:06.952967 22671 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:06.952996 22676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:06.953106 22676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae2722b82784c95ac13bbb3db7f5c40, State: Running, Role: LEADER
I20260927 07:04:06.953194 22676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } }
I20260927 07:04:06.953222 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:06.954495 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } } }
I20260927 07:04:06.954592 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:06.954782 22678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae2722b82784c95ac13bbb3db7f5c40. Latest consensus state: current_term: 1 leader_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae2722b82784c95ac13bbb3db7f5c40" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34793 } } }
I20260927 07:04:06.955047 22678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40 [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:06.955278 22691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:06.955338 22691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:06.955405 22692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:06.955616 22692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:04:06 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:06.957264 22692 catalog_manager.cc:1347] Generated new cluster ID: 6ea7b2c2f5af42edb4e3187e14620df7
I20260927 07:04:06.957312 22692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:06.960621 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:06.948964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.21.2.126:34793)
0927 07:04:06.949171 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:06.949181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:06.949217 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:04:06.949806 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:06.949815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:06.949878 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:06.950029 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:06.950655 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:06.952451 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:06.953478 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:06.954121 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:06.954197 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:06.957308 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:06.957312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:06.957312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:06.957323 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:06.959060 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:06.959477 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:06.959480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:06.959480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:06.959570 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:06.959802 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:06.959803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:06.959803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:06.960033 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:06.960204 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:06.960276 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 07:04:06.960415 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260927 07:04:06.964533 22692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:06.965025 22692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:06.971994 22692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Generated new TSK 0
I20260927 07:04:06.972201 22692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:07.000538 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:07.081427 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:07.081691 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:07.081735 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:07.082922 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:07.082996 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:07.083034 22697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:07.083070 22697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:07.084568 22697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:07.084851 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:07.085093 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:07.090612 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:07.090612 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:07.090675 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:07.091006 22697 server_base.cc:1034] running on GCE node
I20260927 07:04:07.091228 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:07.091490 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:07.092711 22697 hybrid_clock.cc:648] HybridClock initialized: now 1790492647092684 us; error 36 us; skew 500 ppm
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:07.095621 22697 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:07.096139 22697 webserver.cc:460] Webserver started at http://127.21.2.65:36785/ using document root <none> and password file <none>
I20260927 07:04:07.096403 22697 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:07.096454 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:07.096648 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:07.097836 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "cdb97d92bdee493a8e79783f822dec61"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "262659602902b2d21b2998fd3f74d305"
server_key_iv: "a37df2fa3a8c434f529b03123516fb7e"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.098225 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "cdb97d92bdee493a8e79783f822dec61"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "262659602902b2d21b2998fd3f74d305"
server_key_iv: "a37df2fa3a8c434f529b03123516fb7e"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.099665 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:07.100445 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.100636 22697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:07.100724 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "cdb97d92bdee493a8e79783f822dec61"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "262659602902b2d21b2998fd3f74d305"
server_key_iv: "a37df2fa3a8c434f529b03123516fb7e"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.100775 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:07.125522 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:07.128130 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:07.128302 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:07.128862 22697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:07.128926 22697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.128957 22697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:07.128974 22697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:07.136178 22697 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:38271
I20260927 07:04:07.136237 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:38271 every 8 connection(s)
I20260927 07:04:07.136538 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:07.139341 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.129207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:33637 (local address 127.21.2.126:34793)
0927 07:04:07.129256 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.129258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.129535 (+ 277us) server_negotiation.cc:407] Connection header received
0927 07:04:07.130002 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.130004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.130018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.130099 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.130882 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.131758 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.133339 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.133715 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.133799 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.136364 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.136366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.136366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.136374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.137904 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.138336 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.138337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.138337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.138401 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.138661 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.138662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.138662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.138784 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.138854 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.139146 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.139249 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:04:07.139524 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.129209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.65:33637)
0927 07:04:07.129409 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.129444 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.129910 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.130147 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.130150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.130501 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.130786 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.130794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.131856 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.131857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.133243 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.133247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.133824 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.133825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.133968 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.134560 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.134569 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.136259 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.138012 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.138019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.138021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.138249 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.138475 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.138476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.138476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.138586 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.138858 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.138864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.138999 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.139115 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.139270 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260927 07:04:07.140048 22828 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34793
I20260927 07:04:07.140161 22828 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:07.140358 22828 heartbeater.cc:511] Master 127.21.2.126:34793 requested a full tablet report, sending...
I20260927 07:04:07.140996 22624 ts_manager.cc:194] Registered new tserver with Master: cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271)
I20260927 07:04:07.141717 22624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:33637
I20260927 07:04:07.145820 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22697
I20260927 07:04:07.145937 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:07.146103 21513 external_mini_cluster.cc:1428] Setting key 0c0c734a032898f83103b2d7155ef92f
I20260927 07:04:07.146929 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.141216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.65:41811)
0927 07:04:07.141350 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.141359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.141434 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.141563 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.141563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.141749 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.141828 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.141830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.142505 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.142506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.143019 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.143020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.143438 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.143439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.143518 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.143977 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.143983 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.144264 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.145870 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.145871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.145871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.146066 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.146284 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.146285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.146285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.146344 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.146593 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.146594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.146638 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.146733 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.146834 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260927 07:04:07.146925 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.141216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:41811 (local address 127.21.2.126:34793)
0927 07:04:07.141253 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.141255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.141418 (+ 163us) server_negotiation.cc:407] Connection header received
0927 07:04:07.141493 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.141494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.141503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.141558 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.141897 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.142412 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.143103 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.143319 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.143444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.144370 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.144371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.144372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.144378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.145769 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.146149 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.146150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.146151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.146205 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.146415 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.146416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.146417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.146527 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.146600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.146758 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.146833 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:07.170272 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:07.258399 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:07.258617 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:07.258682 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:07.259855 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:07.259923 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:07.259958 22837 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:07.259999 22837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:07.261427 22837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:07.261713 22837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:07.261941 22837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:07.266877 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:07.266873 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:07.267052 22837 server_base.cc:1034] running on GCE node
W20260927 07:04:07.266911 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:07.267333 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:07.267669 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:07.268817 22837 hybrid_clock.cc:648] HybridClock initialized: now 1790492647268806 us; error 30 us; skew 500 ppm
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:07.271215 22837 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:07.271668 22837 webserver.cc:460] Webserver started at http://127.21.2.66:44835/ using document root <none> and password file <none>
I20260927 07:04:07.271873 22837 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:07.271914 22837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:07.271996 22837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:07.272964 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "26b6f45be9d045f6a8ce48ec79479922"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "12a95d78a639423826b3a573b22a427b"
server_key_iv: "610734a9861517fff08cc1a228b68da4"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.273252 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "26b6f45be9d045f6a8ce48ec79479922"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "12a95d78a639423826b3a573b22a427b"
server_key_iv: "610734a9861517fff08cc1a228b68da4"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.274498 22837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:07.275194 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.275415 22837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:07.275525 22837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "26b6f45be9d045f6a8ce48ec79479922"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "12a95d78a639423826b3a573b22a427b"
server_key_iv: "610734a9861517fff08cc1a228b68da4"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.275622 22837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:07.294266 22837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:07.296650 22837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:07.296815 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:07.297380 22837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:07.297438 22837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.297479 22837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:07.297508 22837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:07.304536 22837 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:41379
I20260927 07:04:07.304639 22967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:41379 every 8 connection(s)
I20260927 07:04:07.304926 22837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:07.305742 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22837
I20260927 07:04:07.305841 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:07.306043 21513 external_mini_cluster.cc:1428] Setting key 388377528c1368120c998f5998006851
I20260927 07:04:07.308009 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.297836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:37803 (local address 127.21.2.126:34793)
0927 07:04:07.297907 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.297910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.298108 (+ 198us) server_negotiation.cc:407] Connection header received
0927 07:04:07.298577 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.298579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.298592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.298678 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.299591 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.300127 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.301723 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.302059 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.302137 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.304968 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.304969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.304970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.304978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.306587 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.307000 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.307001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.307002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.307060 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.307332 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.307333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.307333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.307480 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.307557 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.307834 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.307915 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:04:07.308216 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.297811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.66:37803)
0927 07:04:07.297961 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.298001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.298477 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.298729 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.298732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.299129 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.299445 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.299451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.300233 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.300234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.301634 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.301637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.302172 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.302174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.302344 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.302873 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.302885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.304868 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.306681 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.306688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.306691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.306920 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.307143 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.307144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.307145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.307258 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.307546 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.307553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.307701 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.307812 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.307977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260927 07:04:07.308688 22968 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34793
I20260927 07:04:07.308790 22968 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:07.308988 22968 heartbeater.cc:511] Master 127.21.2.126:34793 requested a full tablet report, sending...
I20260927 07:04:07.309404 22624 ts_manager.cc:194] Registered new tserver with Master: 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379)
I20260927 07:04:07.309798 22624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:37803
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
I20260927 07:04:07.317701 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.311646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:55999 (local address 127.21.2.126:34793)
0927 07:04:07.311694 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.311696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.311792 (+ 96us) server_negotiation.cc:407] Connection header received
0927 07:04:07.311820 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.311821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.311833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.311904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.312185 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.313006 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.313720 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.314035 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.314114 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.315108 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.315109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.315109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.315118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.316718 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.317013 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.317014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.317014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.317062 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.317211 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.317212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.317212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.317400 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.317454 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.317578 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.317639 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:07.317727 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.311576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.66:55999)
0927 07:04:07.311712 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.311722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.311783 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.311905 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.311906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.312056 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.312120 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.312122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.313118 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.313119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.313631 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.313633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.314128 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.314129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.314195 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.314727 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.314731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.315013 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.316788 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.316789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.316789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.316950 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.317119 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.317119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.317120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.317173 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.317469 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.317469 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.317507 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.317559 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.317635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:07.331847 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:07.411938 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:07.412135 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:07.412166 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:07.413349 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:07.413390 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:07.413410 22977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:07.413429 22977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:07.414893 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:07.415149 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:07.415413 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:07.420166 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:07.420154 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:07.420321 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:07.420459 22977 server_base.cc:1034] running on GCE node
I20260927 07:04:07.420645 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:07.420833 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:07.421964 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790492647421945 us; error 34 us; skew 500 ppm
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:07.424394 22977 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:07.424816 22977 webserver.cc:460] Webserver started at http://127.21.2.67:36299/ using document root <none> and password file <none>
I20260927 07:04:07.425050 22977 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:07.425117 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:07.425266 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:07.426282 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "a9c5b89b2bd84112bd53bedd554e3de4"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "b7a96feddc8c38341eecf2042ce19723"
server_key_iv: "e5edd58cc7ff4706bcc301bd2a318619"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.426638 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "a9c5b89b2bd84112bd53bedd554e3de4"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "b7a96feddc8c38341eecf2042ce19723"
server_key_iv: "e5edd58cc7ff4706bcc301bd2a318619"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.427994 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:07.428666 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.428836 22977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:07.428930 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "a9c5b89b2bd84112bd53bedd554e3de4"
format_stamp: "Formatted at 2026-09-27 07:04:07 on dist-test-slave-ltqz"
server_key: "b7a96feddc8c38341eecf2042ce19723"
server_key_iv: "e5edd58cc7ff4706bcc301bd2a318619"
server_key_version: "encryptionkey@0"
I20260927 07:04:07.429008 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:07.448683 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:07.450768 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:07.450946 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:07.451522 22977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:07.451587 22977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.451629 22977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:07.451656 22977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:07.459316 22977 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:42243
I20260927 07:04:07.459728 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:42243 every 8 connection(s)
I20260927 07:04:07.459764 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:07.461725 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.451984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:36425 (local address 127.21.2.126:34793)
0927 07:04:07.452042 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.452044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.452260 (+ 216us) server_negotiation.cc:407] Connection header received
0927 07:04:07.452748 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.452750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.452762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.452846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.453711 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.454576 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.455888 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.456241 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.456321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.458801 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.458802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.458803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.458811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.460515 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.461003 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.461004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.461005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.461056 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.461234 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.461235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.461235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.461332 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.461381 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.461599 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.461658 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:04:07.461925 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.451976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.67:36425)
0927 07:04:07.452129 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.452170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.452651 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.452910 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.452913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.453282 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.453606 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.453613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.454685 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.454686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.455819 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.455822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.456345 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.456346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.456491 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.457016 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.457028 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.458706 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.460618 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.460624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.460626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.460937 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.461097 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.461097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.461097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.461180 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.461387 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.461392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.461497 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.461575 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.461684 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260927 07:04:07.462632 23108 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34793
I20260927 07:04:07.462738 23108 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:07.462915 23108 heartbeater.cc:511] Master 127.21.2.126:34793 requested a full tablet report, sending...
I20260927 07:04:07.463325 22625 ts_manager.cc:194] Registered new tserver with Master: a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67:42243)
I20260927 07:04:07.463871 22625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:36425
I20260927 07:04:07.466615 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 22977
I20260927 07:04:07.466714 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:07.466861 21513 external_mini_cluster.cc:1428] Setting key 9d8345c7f6a6121e34c6d82e06cbbd09
I20260927 07:04:07.467785 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:04:07.469245 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.463651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.21.2.67:49781)
0927 07:04:07.463797 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.463808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.463865 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.464054 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.464055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.464251 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.464325 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.464327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.464935 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.464936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.465405 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.465406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.465697 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.465697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.465752 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.466096 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.466100 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.466357 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.468295 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.468296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.468296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.468458 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.468670 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.468670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.468670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.468731 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.469003 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.469003 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.469039 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.469088 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.469165 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260927 07:04:07.469333 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.463741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:49781 (local address 127.21.2.126:34793)
0927 07:04:07.463785 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.463787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.463852 (+ 65us) server_negotiation.cc:407] Connection header received
0927 07:04:07.463920 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.463921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.463950 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.464005 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.464406 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.464851 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.465471 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.465637 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.465683 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.466467 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.466469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.466469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.466476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.468208 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.468535 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.468536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.468536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.468605 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.468794 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.468795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.468795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.468941 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.469009 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.469115 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.469196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:07.490077 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.483606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.21.2.126:34793)
0927 07:04:07.483665 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.483666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.483676 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:07.483754 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.483755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.483764 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.483813 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.484334 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.484806 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.485493 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.485666 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.485712 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.487964 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.487965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.487966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.487972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.489113 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.489416 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.489417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.489417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.489463 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.489647 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.489648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.489649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.489765 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.489852 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.489936 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.489996 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:04:07.491607 22625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:07.493147 22625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:07.499500 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:42243 (local address 127.0.0.1:39280)
0927 07:04:07.495772 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.495783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.495850 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.496050 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.496053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.496055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.496124 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.496126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497486 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.497487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.498352 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.498354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.499192 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.499193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.499252 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.499264 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.499348 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.499420 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260927 07:04:07.499552 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.21.2.67:42243)
0927 07:04:07.495722 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.495727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.495855 (+ 128us) server_negotiation.cc:407] Connection header received
0927 07:04:07.495917 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.495919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.495950 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.496025 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.496201 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497366 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.498443 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.499114 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.499189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.499254 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.499414 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.499454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260927 07:04:07.499797 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:41379 (local address 127.0.0.1:52678)
0927 07:04:07.495574 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.495617 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.495684 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.496156 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.496157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.496160 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.496227 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.496230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497460 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.497461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.498627 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.498630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.499559 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.499560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.499640 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.499645 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.499680 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.499719 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":141,"threads_started":1}
I20260927 07:04:07.500183 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.21.2.66:41379)
0927 07:04:07.495973 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.495977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.495987 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:07.496026 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.496028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.496044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.496122 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.496306 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497369 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.498717 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.499461 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.499969 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.500029 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.500053 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.500088 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":79,"threads_started":1}
I20260927 07:04:07.500204 23042 tablet_service.cc:1467] Processing CreateTablet for tablet b9934f4761084e678f27e4e710ae5b03 (DEFAULT_TABLE table=test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.500594 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9934f4761084e678f27e4e710ae5b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.500771 22902 tablet_service.cc:1467] Processing CreateTablet for tablet b9934f4761084e678f27e4e710ae5b03 (DEFAULT_TABLE table=test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.500885 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:38271 (local address 127.0.0.1:43810)
0927 07:04:07.495574 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.495616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.495684 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.496244 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.496245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.496247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.496286 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.496288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497302 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.497303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.498630 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.498632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.500655 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.500656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.500724 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.500728 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.500778 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.500819 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":34,"thread_start_us":56,"threads_started":1}
I20260927 07:04:07.501065 22902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9934f4761084e678f27e4e710ae5b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.501133 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.495848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.21.2.65:38271)
0927 07:04:07.496027 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.496032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.496041 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:07.496096 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.496098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.496112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.496184 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.496363 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.497220 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.499306 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.500494 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.500566 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.500622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.500997 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.501045 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260927 07:04:07.501679 22762 tablet_service.cc:1467] Processing CreateTablet for tablet b9934f4761084e678f27e4e710ae5b03 (DEFAULT_TABLE table=test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.501976 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9934f4761084e678f27e4e710ae5b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.504043 23131 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Bootstrap starting.
I20260927 07:04:07.504603 23132 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Bootstrap starting.
I20260927 07:04:07.505143 23132 tablet_bootstrap.cc:654] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.505177 23131 tablet_bootstrap.cc:654] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.505450 23132 log.cc:824] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:07.505511 23131 log.cc:824] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:07.505822 23133 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Bootstrap starting.
I20260927 07:04:07.506170 23132 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: No bootstrap required, opened a new log
I20260927 07:04:07.506224 23132 ts_tablet_manager.cc:1397] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:07.506223 23131 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: No bootstrap required, opened a new log
I20260927 07:04:07.506284 23131 ts_tablet_manager.cc:1397] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:07.507931 23131 raft_consensus.cc:348] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.508019 23133 tablet_bootstrap.cc:654] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.508082 23131 raft_consensus.cc:374] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.508126 23131 raft_consensus.cc:729] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c5b89b2bd84112bd53bedd554e3de4, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.508153 23132 raft_consensus.cc:348] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.508252 23131 consensus_queue.cc:260] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.508325 23132 raft_consensus.cc:374] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.508374 23132 raft_consensus.cc:729] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b6f45be9d045f6a8ce48ec79479922, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.508368 23133 log.cc:824] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:07.508561 23131 ts_tablet_manager.cc:1428] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:04:07.508693 23108 heartbeater.cc:503] Master 127.21.2.126:34793 was elected leader, sending a full tablet report...
I20260927 07:04:07.508946 23132 consensus_queue.cc:260] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.509244 23132 ts_tablet_manager.cc:1428] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:07.509294 22968 heartbeater.cc:503] Master 127.21.2.126:34793 was elected leader, sending a full tablet report...
I20260927 07:04:07.509977 23133 tablet_bootstrap.cc:492] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: No bootstrap required, opened a new log
I20260927 07:04:07.510052 23133 ts_tablet_manager.cc:1397] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 07:04:07.511693 23133 raft_consensus.cc:348] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.511813 23133 raft_consensus.cc:374] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.511834 23133 raft_consensus.cc:729] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb97d92bdee493a8e79783f822dec61, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.511931 23133 consensus_queue.cc:260] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.512140 23133 ts_tablet_manager.cc:1428] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:07.512204 22828 heartbeater.cc:503] Master 127.21.2.126:34793 was elected leader, sending a full tablet report...
I20260927 07:04:07.521499 23136 raft_consensus.cc:482] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:07.521597 23136 raft_consensus.cc:504] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.521953 23136 leader_election.cc:288] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379), cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271)
I20260927 07:04:07.525383 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.522129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:41379 (local address 127.21.2.67:40751)
0927 07:04:07.522298 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.522306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.522380 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.522523 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.522524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.522525 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.522617 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.522619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.523534 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.523535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.524683 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.524686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.525173 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.525174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.525235 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.525239 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.525287 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.525319 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260927 07:04:07.525422 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.522194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:40751 (local address 127.21.2.66:41379)
0927 07:04:07.522235 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.522237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.522364 (+ 127us) server_negotiation.cc:407] Connection header received
0927 07:04:07.522441 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.522442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.522453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.522520 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.522734 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.523461 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.524766 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.525100 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.525154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.525161 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.525316 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.525354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:07.525714 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9934f4761084e678f27e4e710ae5b03" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b6f45be9d045f6a8ce48ec79479922" is_pre_election: true
I20260927 07:04:07.525862 22922 raft_consensus.cc:2454] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 0.
I20260927 07:04:07.526078 22995 leader_election.cc:302] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, a9c5b89b2bd84112bd53bedd554e3de4; no voters:
I20260927 07:04:07.526146 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.522161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48069 (local address 127.21.2.65:38271)
0927 07:04:07.522210 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.522212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.522305 (+ 93us) server_negotiation.cc:407] Connection header received
0927 07:04:07.522360 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.522361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.522372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.522445 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.522620 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.523336 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.525190 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.525712 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.525805 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.525814 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.526032 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.526072 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:04:07.526078 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.522110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:38271 (local address 127.21.2.67:48069)
0927 07:04:07.522224 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.522237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.522298 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.522447 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.522447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.522449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.522520 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.522522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.523444 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.523445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.524683 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.524686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.525808 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.525809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.525902 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.525905 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.525971 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.526012 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":32}
I20260927 07:04:07.526257 23136 raft_consensus.cc:2781] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:07.526327 23136 raft_consensus.cc:482] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:07.526352 23136 raft_consensus.cc:3037] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.526518 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9934f4761084e678f27e4e710ae5b03" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61" is_pre_election: true
I20260927 07:04:07.526650 22782 raft_consensus.cc:2454] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 0.
I20260927 07:04:07.527189 23136 raft_consensus.cc:504] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.527333 23136 leader_election.cc:288] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 election: Requested vote from peers 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379), cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271)
I20260927 07:04:07.527474 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9934f4761084e678f27e4e710ae5b03" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b6f45be9d045f6a8ce48ec79479922"
I20260927 07:04:07.527495 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9934f4761084e678f27e4e710ae5b03" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61"
I20260927 07:04:07.527560 22782 raft_consensus.cc:3037] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.527560 22922 raft_consensus.cc:3037] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.528414 22782 raft_consensus.cc:2454] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 1.
I20260927 07:04:07.528414 22922 raft_consensus.cc:2454] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 1.
I20260927 07:04:07.528570 22995 leader_election.cc:302] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, a9c5b89b2bd84112bd53bedd554e3de4; no voters:
I20260927 07:04:07.528664 23136 raft_consensus.cc:2781] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:07.528826 23136 raft_consensus.cc:686] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 LEADER]: Becoming Leader. State: Replica: a9c5b89b2bd84112bd53bedd554e3de4, State: Running, Role: LEADER
I20260927 07:04:07.528915 23136 consensus_queue.cc:237] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } }
I20260927 07:04:07.529686 22625 catalog_manager.cc:5495] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:07.536134 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.533621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.21.2.67:42243)
0927 07:04:07.533672 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.533674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.533683 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:07.533761 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.533761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.533778 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.533831 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:07.533955 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.534414 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.535209 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.535489 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.535561 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.535775 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.535875 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.535964 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.536018 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.536038 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 07:04:07.538043 22922 raft_consensus.cc:1261] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Refusing update from remote peer a9c5b89b2bd84112bd53bedd554e3de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.538084 22782 raft_consensus.cc:1261] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Refusing update from remote peer a9c5b89b2bd84112bd53bedd554e3de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.538375 23141 consensus_queue.cc:1035] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.538535 23136 consensus_queue.cc:1035] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.540997 23144 mvcc.cc:205] Tried to move back new op lower bound from 7333857884313202688 to 7333857884279050240. Current Snapshot: MvccSnapshot[applied={T|T < 7333857884313202688}]
I20260927 07:04:07.542255 23146 mvcc.cc:205] Tried to move back new op lower bound from 7333857884313202688 to 7333857884279050240. Current Snapshot: MvccSnapshot[applied={T|T < 7333857884313202688}]
I20260927 07:04:07.544435 22625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:07.544512 22625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:07.545264 22625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Sending DeleteTablet for 3 replicas of tablet b9934f4761084e678f27e4e710ae5b03
I20260927 07:04:07.545610 22762 tablet_service.cc:1514] Processing DeleteTablet for tablet b9934f4761084e678f27e4e710ae5b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:43810
I20260927 07:04:07.545710 22902 tablet_service.cc:1514] Processing DeleteTablet for tablet b9934f4761084e678f27e4e710ae5b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:52678
I20260927 07:04:07.545825 23042 tablet_service.cc:1514] Processing DeleteTablet for tablet b9934f4761084e678f27e4e710ae5b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:39280
I20260927 07:04:07.545922 22625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 07:04:07.545950 23154 tablet_replica.cc:330] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: stopping tablet replica
W20260927 07:04:07.546075 22625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:07.546180 23156 tablet_replica.cc:330] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: stopping tablet replica
I20260927 07:04:07.546268 23154 raft_consensus.cc:2229] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:07.546289 23156 raft_consensus.cc:2229] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:07.546442 23156 raft_consensus.cc:2258] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.547096 23156 ts_tablet_manager.cc:1905] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.547384 22762 tablet_service.cc:1467] Processing CreateTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (DEFAULT_TABLE table=test-table [id=7f658e0921914c16bd387956e3575091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.547438 23042 tablet_service.cc:1467] Processing CreateTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (DEFAULT_TABLE table=test-table [id=7f658e0921914c16bd387956e3575091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.547510 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77b8ec6a5cd477fb8538e4f2f636fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.547547 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77b8ec6a5cd477fb8538e4f2f636fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.548933 23156 ts_tablet_manager.cc:1918] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:07.548942 23133 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Bootstrap starting.
I20260927 07:04:07.549036 23156 log.cc:1196] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/b9934f4761084e678f27e4e710ae5b03
I20260927 07:04:07.549124 22902 tablet_service.cc:1467] Processing CreateTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (DEFAULT_TABLE table=test-table [id=7f658e0921914c16bd387956e3575091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:07.549238 22902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77b8ec6a5cd477fb8538e4f2f636fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.549356 23133 tablet_bootstrap.cc:654] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.549383 23156 ts_tablet_manager.cc:1939] T b9934f4761084e678f27e4e710ae5b03 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting consensus metadata
I20260927 07:04:07.550526 23132 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Bootstrap starting.
I20260927 07:04:07.550700 22610 catalog_manager.cc:4841] TS a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67:42243): tablet b9934f4761084e678f27e4e710ae5b03 (table test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]) successfully deleted
I20260927 07:04:07.550850 23131 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Bootstrap starting.
I20260927 07:04:07.551199 23131 tablet_bootstrap.cc:654] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.551218 23132 tablet_bootstrap.cc:654] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.551615 23154 raft_consensus.cc:2258] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.551837 23155 tablet_replica.cc:330] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: stopping tablet replica
I20260927 07:04:07.551928 23132 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: No bootstrap required, opened a new log
I20260927 07:04:07.552044 23132 ts_tablet_manager.cc:1397] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:04:07.552171 23155 raft_consensus.cc:2229] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:07.552222 23132 raft_consensus.cc:348] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.552296 23132 raft_consensus.cc:374] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.552326 23132 raft_consensus.cc:729] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b6f45be9d045f6a8ce48ec79479922, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.552377 23155 raft_consensus.cc:2258] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.552459 23154 ts_tablet_manager.cc:1905] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.552533 23131 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: No bootstrap required, opened a new log
I20260927 07:04:07.552582 23131 ts_tablet_manager.cc:1397] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:07.552732 23131 raft_consensus.cc:348] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.552826 23131 raft_consensus.cc:374] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.552855 23131 raft_consensus.cc:729] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c5b89b2bd84112bd53bedd554e3de4, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.552901 23131 consensus_queue.cc:260] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.552995 23131 ts_tablet_manager.cc:1428] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.552377 23132 consensus_queue.cc:260] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.553385 23132 ts_tablet_manager.cc:1428] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:04:07.553982 23154 ts_tablet_manager.cc:1918] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:07.553989 23155 ts_tablet_manager.cc:1905] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.554067 23154 log.cc:1196] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/b9934f4761084e678f27e4e710ae5b03
I20260927 07:04:07.554342 23154 ts_tablet_manager.cc:1939] T b9934f4761084e678f27e4e710ae5b03 P cdb97d92bdee493a8e79783f822dec61: Deleting consensus metadata
I20260927 07:04:07.554767 22612 catalog_manager.cc:4841] TS cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271): tablet b9934f4761084e678f27e4e710ae5b03 (table test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]) successfully deleted
W20260927 07:04:07.555474 22969 tablet.cc:2369] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:07.555689 23155 ts_tablet_manager.cc:1918] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:07.555766 23155 log.cc:1196] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/b9934f4761084e678f27e4e710ae5b03
I20260927 07:04:07.556010 23155 ts_tablet_manager.cc:1939] T b9934f4761084e678f27e4e710ae5b03 P 26b6f45be9d045f6a8ce48ec79479922: Deleting consensus metadata
I20260927 07:04:07.556413 22611 catalog_manager.cc:4841] TS 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379): tablet b9934f4761084e678f27e4e710ae5b03 (table test-table [id=368a693abdbf4aa8992f8e7dd8bcbaa8]) successfully deleted
I20260927 07:04:07.559820 23133 tablet_bootstrap.cc:492] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: No bootstrap required, opened a new log
I20260927 07:04:07.559868 23133 ts_tablet_manager.cc:1397] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260927 07:04:07.560038 23133 raft_consensus.cc:348] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.560101 23133 raft_consensus.cc:374] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.560114 23133 raft_consensus.cc:729] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb97d92bdee493a8e79783f822dec61, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.560169 23133 consensus_queue.cc:260] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.560293 23133 ts_tablet_manager.cc:1428] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.620667 23138 raft_consensus.cc:482] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:07.620772 23138 raft_consensus.cc:504] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.621093 23138 leader_election.cc:288] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271), a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67:42243)
I20260927 07:04:07.624698 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.621284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:38271 (local address 127.21.2.66:47883)
0927 07:04:07.621347 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.621360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.621411 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.621599 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.621600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.621601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.621671 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.621673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.622635 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.622636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.623838 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.623841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.624411 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.624412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.624493 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.624496 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.624565 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.624608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":15}
I20260927 07:04:07.624682 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.621369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47883 (local address 127.21.2.65:38271)
0927 07:04:07.621413 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.621415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.621425 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:07.621490 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.621491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.621503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.621551 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.621769 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.622529 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.623963 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.624325 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.624411 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.624422 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.624570 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.624610 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:07.624943 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77b8ec6a5cd477fb8538e4f2f636fc9" candidate_uuid: "26b6f45be9d045f6a8ce48ec79479922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61" is_pre_election: true
I20260927 07:04:07.625051 22782 raft_consensus.cc:2454] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b6f45be9d045f6a8ce48ec79479922 in term 0.
I20260927 07:04:07.625056 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.621463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:40857 (local address 127.21.2.67:42243)
0927 07:04:07.621509 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.621512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.621524 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:04:07.621619 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.621620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.621629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.621662 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:07.621868 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.622452 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.623954 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.624494 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.624586 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.624597 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.624944 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.624984 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:07.625241 22856 leader_election.cc:302] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, cdb97d92bdee493a8e79783f822dec61; no voters:
I20260927 07:04:07.625378 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.621298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:42243 (local address 127.21.2.66:40857)
0927 07:04:07.621476 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.621486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.621529 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.621723 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.621723 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.621725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:07.621784 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.621785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.622539 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.622541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.623845 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.623846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.624638 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.624639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.624720 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.624724 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.625174 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.625221 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":53,"thread_start_us":65,"threads_started":1}
I20260927 07:04:07.625394 23138 raft_consensus.cc:2781] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:07.625460 23138 raft_consensus.cc:482] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:07.625486 23138 raft_consensus.cc:3037] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.625677 23062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77b8ec6a5cd477fb8538e4f2f636fc9" candidate_uuid: "26b6f45be9d045f6a8ce48ec79479922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" is_pre_election: true
I20260927 07:04:07.625802 23062 raft_consensus.cc:2454] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b6f45be9d045f6a8ce48ec79479922 in term 0.
I20260927 07:04:07.626317 23138 raft_consensus.cc:504] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.626458 23138 leader_election.cc:288] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [CANDIDATE]: Term 1 election: Requested vote from peers cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271), a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67:42243)
I20260927 07:04:07.626618 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77b8ec6a5cd477fb8538e4f2f636fc9" candidate_uuid: "26b6f45be9d045f6a8ce48ec79479922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61"
I20260927 07:04:07.626655 23062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77b8ec6a5cd477fb8538e4f2f636fc9" candidate_uuid: "26b6f45be9d045f6a8ce48ec79479922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c5b89b2bd84112bd53bedd554e3de4"
I20260927 07:04:07.626716 22782 raft_consensus.cc:3037] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.626734 23062 raft_consensus.cc:3037] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.627286 22782 raft_consensus.cc:2454] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b6f45be9d045f6a8ce48ec79479922 in term 1.
I20260927 07:04:07.627475 22856 leader_election.cc:302] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, cdb97d92bdee493a8e79783f822dec61; no voters:
I20260927 07:04:07.627578 23138 raft_consensus.cc:2781] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:07.627648 23062 raft_consensus.cc:2454] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b6f45be9d045f6a8ce48ec79479922 in term 1.
I20260927 07:04:07.627684 23138 raft_consensus.cc:686] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 LEADER]: Becoming Leader. State: Replica: 26b6f45be9d045f6a8ce48ec79479922, State: Running, Role: LEADER
I20260927 07:04:07.627758 23138 consensus_queue.cc:237] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.628450 22624 catalog_manager.cc:5495] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "26b6f45be9d045f6a8ce48ec79479922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } health_report { overall_health: UNKNOWN } } }
W20260927 07:04:07.637368 22829 tablet.cc:2369] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492646, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:07.667150 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.660094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34793 (local address 127.0.0.1:48594)
0927 07:04:07.660245 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.660264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.660323 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.660463 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.660464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.660737 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:07.660793 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.660796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.661653 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.661653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.662171 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.662172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.662632 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.662633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.662699 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.663167 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:07.663172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:07.664607 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:07.666153 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.666159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.666160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.666339 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.666591 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:07.666592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:07.666592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:07.666647 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:07.666874 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:07.666879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:07.666917 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.666968 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.667055 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260927 07:04:07.667200 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.660163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.21.2.126:34793)
0927 07:04:07.660215 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.660217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.660333 (+ 116us) server_negotiation.cc:407] Connection header received
0927 07:04:07.660364 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.660365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.660376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.660451 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.660871 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.661581 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.662251 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.662555 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.662628 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.664694 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.664696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.664696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.664702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.666086 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.666456 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.666458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.666458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.666517 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.666705 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.666706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.666706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.666817 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.666868 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.667001 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.667103 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":29}
I20260927 07:04:07.669080 22624 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:04:07.669255 22624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:07.670619 23042 tablet_service.cc:1467] Processing CreateTablet for tablet 67ba858aaa8c467f829782a454625b4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc3711acf824932b9c3536fbdfcafc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:07.670614 22902 tablet_service.cc:1467] Processing CreateTablet for tablet 67ba858aaa8c467f829782a454625b4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc3711acf824932b9c3536fbdfcafc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:07.670643 22762 tablet_service.cc:1467] Processing CreateTablet for tablet 67ba858aaa8c467f829782a454625b4b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc3711acf824932b9c3536fbdfcafc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:07.670774 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ba858aaa8c467f829782a454625b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.670780 22902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ba858aaa8c467f829782a454625b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.670799 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ba858aaa8c467f829782a454625b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:07.672045 23133 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: Bootstrap starting.
I20260927 07:04:07.672222 23132 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: Bootstrap starting.
I20260927 07:04:07.672242 23131 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: Bootstrap starting.
I20260927 07:04:07.672467 23133 tablet_bootstrap.cc:654] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.672513 23131 tablet_bootstrap.cc:654] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.672621 23132 tablet_bootstrap.cc:654] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:07.673032 23131 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: No bootstrap required, opened a new log
I20260927 07:04:07.673084 23131 ts_tablet_manager.cc:1397] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:04:07.673213 23131 raft_consensus.cc:348] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673285 23132 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: No bootstrap required, opened a new log
I20260927 07:04:07.673293 23131 raft_consensus.cc:374] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.673321 23131 raft_consensus.cc:729] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c5b89b2bd84112bd53bedd554e3de4, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.673334 23132 ts_tablet_manager.cc:1397] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:07.673367 23131 consensus_queue.cc:260] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673391 23133 tablet_bootstrap.cc:492] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: No bootstrap required, opened a new log
I20260927 07:04:07.673444 23133 ts_tablet_manager.cc:1397] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:07.673472 23131 ts_tablet_manager.cc:1428] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:07.673477 23132 raft_consensus.cc:348] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673537 23132 raft_consensus.cc:374] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.673558 23132 raft_consensus.cc:729] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b6f45be9d045f6a8ce48ec79479922, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.673522 23151 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.673602 23132 consensus_queue.cc:260] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673614 23151 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.673588 23133 raft_consensus.cc:348] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673650 23133 raft_consensus.cc:374] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:07.673671 23133 raft_consensus.cc:729] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb97d92bdee493a8e79783f822dec61, State: Initialized, Role: FOLLOWER
I20260927 07:04:07.673686 23132 ts_tablet_manager.cc:1428] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:07.673709 23133 consensus_queue.cc:260] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.673717 23138 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.673784 23138 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.673815 23133 ts_tablet_manager.cc:1428] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:07.673949 23139 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.674029 23139 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 07:04:07.710404 23109 tablet.cc:2369] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:07.807897 23151 raft_consensus.cc:482] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:07.808001 23151 raft_consensus.cc:504] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.808202 23151 leader_election.cc:288] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271), 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379)
I20260927 07:04:07.808473 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ba858aaa8c467f829782a454625b4b" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b6f45be9d045f6a8ce48ec79479922" is_pre_election: true
I20260927 07:04:07.808566 22922 raft_consensus.cc:2454] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 0.
I20260927 07:04:07.808539 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ba858aaa8c467f829782a454625b4b" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61" is_pre_election: true
I20260927 07:04:07.808660 22782 raft_consensus.cc:2454] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 0.
I20260927 07:04:07.808759 22995 leader_election.cc:302] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, a9c5b89b2bd84112bd53bedd554e3de4; no voters:
I20260927 07:04:07.808851 23151 raft_consensus.cc:2781] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:07.808890 23151 raft_consensus.cc:482] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:07.808947 23151 raft_consensus.cc:3037] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.809739 23151 raft_consensus.cc:504] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.809881 23151 leader_election.cc:288] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 election: Requested vote from peers cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271), 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379)
I20260927 07:04:07.810034 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ba858aaa8c467f829782a454625b4b" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb97d92bdee493a8e79783f822dec61"
I20260927 07:04:07.810060 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ba858aaa8c467f829782a454625b4b" candidate_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b6f45be9d045f6a8ce48ec79479922"
I20260927 07:04:07.810132 22922 raft_consensus.cc:3037] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.810137 22782 raft_consensus.cc:3037] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:07.810719 22922 raft_consensus.cc:2454] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 1.
I20260927 07:04:07.810827 22782 raft_consensus.cc:2454] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c5b89b2bd84112bd53bedd554e3de4 in term 1.
I20260927 07:04:07.810901 22995 leader_election.cc:302] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b6f45be9d045f6a8ce48ec79479922, a9c5b89b2bd84112bd53bedd554e3de4; no voters:
I20260927 07:04:07.811010 23151 raft_consensus.cc:2781] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:07.811084 23151 raft_consensus.cc:686] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 LEADER]: Becoming Leader. State: Replica: a9c5b89b2bd84112bd53bedd554e3de4, State: Running, Role: LEADER
I20260927 07:04:07.811136 23151 consensus_queue.cc:237] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } }
I20260927 07:04:07.811419 23141 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: TxnStatusTablet state changed. Reason: New leader a9c5b89b2bd84112bd53bedd554e3de4. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.811498 23141 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:07.811728 23173 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:04:07.811844 23173 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:04:07.811816 22624 catalog_manager.cc:5495] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } health_report { overall_health: HEALTHY } } }
I20260927 07:04:07.822697 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.820354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.21.2.67:42243)
0927 07:04:07.820401 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.820403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.820412 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:07.820503 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.820504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.820516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.820572 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:07.820727 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.821193 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.821963 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.822159 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.822239 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.822431 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.822464 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.822532 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.822600 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.822621 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:07.822721 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.820292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:42243 (local address 127.0.0.1:39292)
0927 07:04:07.820366 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.820379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.820430 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.820577 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.820577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.820578 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:07.820644 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.820647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.821290 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.821291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.821867 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.821869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.822241 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.822242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.822316 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.822338 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.822533 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.822548 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.822599 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.822648 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 07:04:07.823793 22782 raft_consensus.cc:1261] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Refusing update from remote peer a9c5b89b2bd84112bd53bedd554e3de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.823788 22922 raft_consensus.cc:1261] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Refusing update from remote peer a9c5b89b2bd84112bd53bedd554e3de4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.824043 23151 consensus_queue.cc:1035] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.824134 23141 consensus_queue.cc:1035] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.824525 23138 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: TxnStatusTablet state changed. Reason: New leader a9c5b89b2bd84112bd53bedd554e3de4. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.824620 23138 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.824615 23139 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: TxnStatusTablet state changed. Reason: New leader a9c5b89b2bd84112bd53bedd554e3de4. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.824694 23139 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.825263 23136 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.825366 23136 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:07.825371 23139 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.825441 23139 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P cdb97d92bdee493a8e79783f822dec61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.826329 23141 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.826406 23141 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P a9c5b89b2bd84112bd53bedd554e3de4: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:07.826499 23145 mvcc.cc:205] Tried to move back new op lower bound from 7333857885484482560 to 7333857885434626048. Current Snapshot: MvccSnapshot[applied={T|T < 7333857885484482560}]
I20260927 07:04:07.826726 23138 tablet_replica.cc:439] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 } } peers { permanent_uuid: "26b6f45be9d045f6a8ce48ec79479922" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 41379 } } peers { permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 } } }
I20260927 07:04:07.826810 23138 tablet_replica.cc:442] T 67ba858aaa8c467f829782a454625b4b P 26b6f45be9d045f6a8ce48ec79479922: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:07.830533 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.827721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.21.2.66:41379)
0927 07:04:07.827762 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.827764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.827772 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:04:07.827830 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.827831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.827848 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.827928 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:07.828005 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.828762 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.829571 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.829885 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.829962 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.830077 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.830213 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.830282 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.830407 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.830426 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:07.835598 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.832972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:42243 (local address 127.21.2.66:47231)
0927 07:04:07.833167 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.833181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.833251 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.833394 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.833394 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.833395 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:07.833457 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.833459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.834093 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.834094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.834681 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.834683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.835072 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.835073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.835153 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.835167 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.835407 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.835409 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.835469 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.835514 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260927 07:04:07.835583 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.833124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47231 (local address 127.21.2.67:42243)
0927 07:04:07.833191 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.833194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.833233 (+ 39us) server_negotiation.cc:407] Connection header received
0927 07:04:07.833317 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.833318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.833330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.833392 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:07.833557 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.834003 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.834789 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.834987 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.835073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.835266 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.835306 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.835404 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.835474 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.835495 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:04:07.838027 22782 raft_consensus.cc:1261] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Refusing update from remote peer 26b6f45be9d045f6a8ce48ec79479922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.838047 23062 raft_consensus.cc:1261] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Refusing update from remote peer 26b6f45be9d045f6a8ce48ec79479922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:07.838521 23161 consensus_queue.cc:1035] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb97d92bdee493a8e79783f822dec61" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 38271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.838716 23161 consensus_queue.cc:1035] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c5b89b2bd84112bd53bedd554e3de4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:07.847877 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.845556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:42451 (local address 127.21.2.66:41379)
0927 07:04:07.845600 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.845601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.845610 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:07.845633 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.845634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.845647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.845706 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:07.845822 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.846300 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.847061 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.847335 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.847441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.847653 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.847681 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.847727 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.847805 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.847824 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:07.847920 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.845318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:41379 (local address 127.21.2.67:42451)
0927 07:04:07.845390 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:07.845408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:07.845466 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:07.845709 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:07.845710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:07.845712 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:07.845757 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.845759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.846396 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.846397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.846966 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:07.846968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.847482 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.847482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:07.847553 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.847570 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:07.847750 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:07.847751 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:04:07.847802 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:04:07.847845 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:04:07.858017 22625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:07.858130 22625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:07.858804 22625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae2722b82784c95ac13bbb3db7f5c40: Sending DeleteTablet for 3 replicas of tablet f77b8ec6a5cd477fb8538e4f2f636fc9
I20260927 07:04:07.859040 22762 tablet_service.cc:1514] Processing DeleteTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:43810
I20260927 07:04:07.859083 23042 tablet_service.cc:1514] Processing DeleteTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:39280
I20260927 07:04:07.859136 23154 tablet_replica.cc:330] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: stopping tablet replica
I20260927 07:04:07.859102 22902 tablet_service.cc:1514] Processing DeleteTablet for tablet f77b8ec6a5cd477fb8538e4f2f636fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:07 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:52678
I20260927 07:04:07.859177 23154 raft_consensus.cc:2229] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:07.859216 23154 raft_consensus.cc:2258] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.859213 23155 tablet_replica.cc:330] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: stopping tablet replica
I20260927 07:04:07.859215 23156 tablet_replica.cc:330] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: stopping tablet replica
I20260927 07:04:07.859269 23155 raft_consensus.cc:2229] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:07.859306 23156 raft_consensus.cc:2229] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:07.859350 23155 raft_consensus.cc:2258] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.859391 23156 raft_consensus.cc:2258] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:07.859575 23154 ts_tablet_manager.cc:1905] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.859902 23156 ts_tablet_manager.cc:1905] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.861191 23154 ts_tablet_manager.cc:1918] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:07.861248 23154 log.cc:1196] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/f77b8ec6a5cd477fb8538e4f2f636fc9
I20260927 07:04:07.861460 23154 ts_tablet_manager.cc:1939] T f77b8ec6a5cd477fb8538e4f2f636fc9 P cdb97d92bdee493a8e79783f822dec61: Deleting consensus metadata
I20260927 07:04:07.861526 23156 ts_tablet_manager.cc:1918] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:07.861577 23156 log.cc:1196] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/f77b8ec6a5cd477fb8538e4f2f636fc9
I20260927 07:04:07.861755 22612 catalog_manager.cc:4841] TS cdb97d92bdee493a8e79783f822dec61 (127.21.2.65:38271): tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (table test-table [id=7f658e0921914c16bd387956e3575091]) successfully deleted
I20260927 07:04:07.861783 23156 ts_tablet_manager.cc:1939] T f77b8ec6a5cd477fb8538e4f2f636fc9 P a9c5b89b2bd84112bd53bedd554e3de4: Deleting consensus metadata
I20260927 07:04:07.862109 22610 catalog_manager.cc:4841] TS a9c5b89b2bd84112bd53bedd554e3de4 (127.21.2.67:42243): tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (table test-table [id=7f658e0921914c16bd387956e3575091]) successfully deleted
I20260927 07:04:07.862190 23155 ts_tablet_manager.cc:1905] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:07.863821 23155 ts_tablet_manager.cc:1918] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:07.863888 23155 log.cc:1196] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/f77b8ec6a5cd477fb8538e4f2f636fc9
I20260927 07:04:07.864100 23155 ts_tablet_manager.cc:1939] T f77b8ec6a5cd477fb8538e4f2f636fc9 P 26b6f45be9d045f6a8ce48ec79479922: Deleting consensus metadata
I20260927 07:04:07.864547 22611 catalog_manager.cc:4841] TS 26b6f45be9d045f6a8ce48ec79479922 (127.21.2.66:41379): tablet f77b8ec6a5cd477fb8538e4f2f636fc9 (table test-table [id=7f658e0921914c16bd387956e3575091]) successfully deleted
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:07.877851 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.867718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.21.2.65:38271)
0927 07:04:07.867773 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.867776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.867934 (+ 158us) server_negotiation.cc:407] Connection header received
0927 07:04:07.867984 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.867985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.867997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.868064 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.868721 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.869508 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.870610 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.870926 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.871000 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.874574 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.874586 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.874586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.874594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.876588 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.876995 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.876999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.876999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.877063 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.877277 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.877277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.877278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.877447 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.877629 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.877669 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.877752 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 07:04:07.878398 22822 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822
I20260927 07:04:07.886971 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:07.879354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.21.2.126:34793)
0927 07:04:07.879416 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:07.879418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:07.879639 (+ 221us) server_negotiation.cc:407] Connection header received
0927 07:04:07.879690 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:07.879691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:07.879703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:07.879766 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:07.880382 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.881134 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.882130 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:07.882422 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:07.882492 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:07.884107 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:07.884109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:07.884109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:07.884117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:07.885742 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.886145 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.886146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.886147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.886216 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:07.886438 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:07.886439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:07.886439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:07.886566 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:07.886667 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:07.886811 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:04:07.886885 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 07:04:07.887444 22625 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48602
I20260927 07:04:07.888150 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22697
I20260927 07:04:07.893464 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22837
I20260927 07:04:07.899349 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22977
I20260927 07:04:07.905085 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 22593
2026-09-27T07:04:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2228 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[23193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[23193](info): set up 2 sockets
Sep 27 07:04:07 dist-test-slave-ltqz krb5kdc[23193](info): commencing operation
krb5kdc: starting...
W20260927 07:04:08.939059 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:09.053591 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40983
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:09.133093 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:09.133285 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:09.133307 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:09.134395 23206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:09.134429 23206 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:04:09.134443 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:09.134456 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:09.134469 23206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:09.134480 23206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:09.135921 23206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:09.136101 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:09.136266 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:09.141012 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.141036 23211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.141036 23212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:09.141426 23206 server_base.cc:1034] running on GCE node
I20260927 07:04:09.141644 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:09.141887 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:09.143065 23206 hybrid_clock.cc:648] HybridClock initialized: now 1790492649143042 us; error 41 us; skew 500 ppm
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:09.145678 23206 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:09.146129 23206 webserver.cc:460] Webserver started at http://127.21.2.126:36977/ using document root <none> and password file <none>
I20260927 07:04:09.146380 23206 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:09.146426 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:09.146572 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:09.147713 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "72fc7f8aff504da6809b20881ef84142"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "b6b14c860d50e620ed0b1d821f32ed51"
server_key_iv: "9d6e5a2ba95138adf16677d5c654234f"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.148080 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "72fc7f8aff504da6809b20881ef84142"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "b6b14c860d50e620ed0b1d821f32ed51"
server_key_iv: "9d6e5a2ba95138adf16677d5c654234f"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.149391 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:09.150180 23221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.150370 23206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:09.150470 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "72fc7f8aff504da6809b20881ef84142"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "b6b14c860d50e620ed0b1d821f32ed51"
server_key_iv: "9d6e5a2ba95138adf16677d5c654234f"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.150547 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:09.166282 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:09.168256 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:09.168442 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:09.173032 23206 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40983
I20260927 07:04:09.173090 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40983 every 8 connection(s)
I20260927 07:04:09.173400 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:09.174002 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.176323 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Bootstrap starting.
I20260927 07:04:09.176918 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.177212 23284 log.cc:824] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:09.177911 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: No bootstrap required, opened a new log
I20260927 07:04:09.178526 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 23206
I20260927 07:04:09.178627 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:09.178790 21513 external_mini_cluster.cc:1428] Setting key 9c9b66ac277acc0ac72137a83518c77b
I20260927 07:04:09.180505 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } }
I20260927 07:04:09.180694 23284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.180749 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72fc7f8aff504da6809b20881ef84142, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.180917 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } }
I20260927 07:04:09.180984 23284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:09.181006 23284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:09.181069 23284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:09.181800 23284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } }
I20260927 07:04:09.181901 23284 leader_election.cc:302] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72fc7f8aff504da6809b20881ef84142; no voters:
I20260927 07:04:09.182097 23284 leader_election.cc:288] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:09.182125 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:09.182241 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [term 1 LEADER]: Becoming Leader. State: Replica: 72fc7f8aff504da6809b20881ef84142, State: Running, Role: LEADER
I20260927 07:04:09.182322 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } }
I20260927 07:04:09.182367 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:09.182638 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72fc7f8aff504da6809b20881ef84142. Latest consensus state: current_term: 1 leader_uuid: "72fc7f8aff504da6809b20881ef84142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } } }
I20260927 07:04:09.182725 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:09.182650 23291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72fc7f8aff504da6809b20881ef84142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fc7f8aff504da6809b20881ef84142" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40983 } } }
I20260927 07:04:09.182770 23291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:09.183095 23294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:09.183662 23294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:09.185400 23294 catalog_manager.cc:1347] Generated new cluster ID: 171c1540fa1744dc85b8044b8eded679
I20260927 07:04:09.185459 23294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.191422 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.179316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.21.2.126:40983)
0927 07:04:09.179514 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.179530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.179554 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:04:09.180490 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.180496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.180544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.180681 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.181458 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.182620 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.183757 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.184285 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.184401 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.187206 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.187211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.187211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.187222 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.189747 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.190160 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.190163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.190163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.190273 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.190482 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.190483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.190483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.190775 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.190924 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.191040 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.191162 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260927 07:04:09.199060 23294 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:09.199827 23294 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:09.208643 23294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Generated new TSK 0
I20260927 07:04:09.208860 23294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:09.244242 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40983
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:09.324580 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:09.324791 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:09.324815 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:09.325872 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:09.325907 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:09.325927 23310 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:04:09.325939 23310 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:09.325955 23310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:09.327638 23310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:09.327833 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:09.328003 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:09.334187 23318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.334167 23316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:09.334213 23310 server_base.cc:1034] running on GCE node
W20260927 07:04:09.334169 23315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:09.334733 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:09.334959 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:09.336117 23310 hybrid_clock.cc:648] HybridClock initialized: now 1790492649336105 us; error 28 us; skew 500 ppm
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:09.338645 23310 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:09.339036 23310 webserver.cc:460] Webserver started at http://127.21.2.65:39933/ using document root <none> and password file <none>
I20260927 07:04:09.339241 23310 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:09.339282 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:09.339422 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:09.340425 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "4c2d52f6204547d2bf5a231e76566cce"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "8374b5062b4c094fd2870eab5ab53517"
server_key_iv: "0d5a7794f624f7b8d5cfeb97918a87bb"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.340728 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "4c2d52f6204547d2bf5a231e76566cce"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "8374b5062b4c094fd2870eab5ab53517"
server_key_iv: "0d5a7794f624f7b8d5cfeb97918a87bb"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.342024 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:04:09.342751 23325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.342931 23310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:09.343024 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "4c2d52f6204547d2bf5a231e76566cce"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "8374b5062b4c094fd2870eab5ab53517"
server_key_iv: "0d5a7794f624f7b8d5cfeb97918a87bb"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.343106 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:09.364655 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:09.366897 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:09.367080 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:09.367720 23310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:09.367789 23310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.367831 23310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:09.367859 23310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.374872 23310 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:41993
I20260927 07:04:09.374939 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:41993 every 8 connection(s)
I20260927 07:04:09.375252 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:09.378015 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.368223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:43605 (local address 127.21.2.126:40983)
0927 07:04:09.368266 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.368268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.368502 (+ 234us) server_negotiation.cc:407] Connection header received
0927 07:04:09.369081 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.369083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.369098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.369186 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.370024 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.370905 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.372505 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.372731 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.372821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.375213 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.375215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.375216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.375223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.376775 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.377188 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.377189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.377189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.377243 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.377455 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.377455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.377456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.377559 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.377611 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.377886 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.377953 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:09.378173 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.368221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.65:43605)
0927 07:04:09.368388 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.368422 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.368927 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.369240 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.369245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.369589 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.369915 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.369922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.371027 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.371028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.372400 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.372403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.372842 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.372843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.372977 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.373514 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.373522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.375114 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.376869 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.376875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.376876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.377108 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.377302 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.377302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.377303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.377399 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.377633 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.377638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.377769 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.377860 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.377978 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260927 07:04:09.378654 23441 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40983
I20260927 07:04:09.378731 23441 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:09.378907 23441 heartbeater.cc:511] Master 127.21.2.126:40983 requested a full tablet report, sending...
I20260927 07:04:09.379300 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 23310
I20260927 07:04:09.379415 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:09.379613 21513 external_mini_cluster.cc:1428] Setting key a95e9f2c01662365f8ad2481709f1f3d
I20260927 07:04:09.379648 23237 ts_manager.cc:194] Registered new tserver with Master: 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993)
I20260927 07:04:09.380592 23237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:43605
I20260927 07:04:09.386034 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.379769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:40415 (local address 127.21.2.126:40983)
0927 07:04:09.379810 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.379812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.379989 (+ 177us) server_negotiation.cc:407] Connection header received
0927 07:04:09.380019 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.380020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.380029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.380086 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.380463 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.381095 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.381940 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.382256 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.382337 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.383453 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.383454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.383455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.383461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.384907 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.385255 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.385256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.385257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.385310 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.385520 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.385521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.385522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.385632 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.385705 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.385869 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.385943 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:09.386047 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.379749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.65:40415)
0927 07:04:09.379909 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.379920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.379978 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.380127 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.380135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.380324 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.380386 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.380388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.381186 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.381186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.381842 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.381844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.382365 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.382366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.382452 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.382882 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.382887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.383291 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.384992 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.384993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.384994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.385167 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.385384 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.385385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.385385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.385450 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.385698 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.385699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.385741 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.385844 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.385945 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:09.405920 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40983
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:09.486433 23450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:09.486629 23450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:09.486653 23450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:09.487776 23450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:09.487810 23450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:09.487830 23450 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:04:09.487843 23450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:09.487859 23450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:09.489223 23450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:09.489391 23450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:09.489553 23450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:09.494504 23458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.494498 23456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.494504 23455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:09.494832 23450 server_base.cc:1034] running on GCE node
I20260927 07:04:09.495028 23450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:09.495267 23450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:09.496450 23450 hybrid_clock.cc:648] HybridClock initialized: now 1790492649496439 us; error 33 us; skew 500 ppm
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:09.498844 23450 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:09.499282 23450 webserver.cc:460] Webserver started at http://127.21.2.66:36175/ using document root <none> and password file <none>
I20260927 07:04:09.499516 23450 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:09.499562 23450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:09.499648 23450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:09.500682 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "88b20e3c35864595bf37106859fc85ad"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "d9ab1093beafdc29123c5bc1a4baf381"
server_key_iv: "5340e7908c978ca553d2ff5831cd7688"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.500990 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "88b20e3c35864595bf37106859fc85ad"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "d9ab1093beafdc29123c5bc1a4baf381"
server_key_iv: "5340e7908c978ca553d2ff5831cd7688"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.502249 23450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:09.502928 23465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.503098 23450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:09.503201 23450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "88b20e3c35864595bf37106859fc85ad"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "d9ab1093beafdc29123c5bc1a4baf381"
server_key_iv: "5340e7908c978ca553d2ff5831cd7688"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.503278 23450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:09.520181 23450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:09.522037 23450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:09.522195 23450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:09.522737 23450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:09.522801 23450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.522840 23450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:09.522866 23450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.530345 23450 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:37267
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.530449 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:37267 every 8 connection(s)
I20260927 07:04:09.530740 23450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:09.533697 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.523127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:39563 (local address 127.21.2.126:40983)
0927 07:04:09.523194 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.523197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.523472 (+ 275us) server_negotiation.cc:407] Connection header received
0927 07:04:09.524016 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.524017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.524031 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.524095 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.524803 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.525325 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.526399 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.526596 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.526682 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.530917 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.530919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.530919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.530929 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.532250 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.532668 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.532670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.532670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.532734 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.533021 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.533023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.533023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.533153 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.533230 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.533527 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.533607 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:04:09.533879 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.523116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.66:39563)
0927 07:04:09.523292 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.523338 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.523908 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.524171 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.524174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.524470 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.524705 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.524710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.525423 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.525424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.526297 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.526299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.526691 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.526692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.526839 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.527455 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.527466 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.530806 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.532354 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.532361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.532362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.532579 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.532815 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.532816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.532816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.532958 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.533220 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.533226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.533385 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.533488 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.533643 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
I20260927 07:04:09.534330 23581 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40983
I20260927 07:04:09.534440 23581 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:09.534607 23581 heartbeater.cc:511] Master 127.21.2.126:40983 requested a full tablet report, sending...
I20260927 07:04:09.535017 23238 ts_manager.cc:194] Registered new tserver with Master: 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267)
I20260927 07:04:09.535593 23238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:39563
I20260927 07:04:09.540727 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.535402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:55173 (local address 127.21.2.126:40983)
0927 07:04:09.535442 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.535444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.535544 (+ 100us) server_negotiation.cc:407] Connection header received
0927 07:04:09.535571 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.535572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.535582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.535650 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.535973 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.536440 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.537138 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.537337 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.537409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.538327 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.538328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.538328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.538335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.539572 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.539917 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.539918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.539919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.539978 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.540172 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.540173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.540173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.540326 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.540405 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.540558 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.540632 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:04:09.540755 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.535322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.66:55173)
0927 07:04:09.535468 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.535478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.535537 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.535674 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.535674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.535846 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.535895 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.535896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.536534 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.536535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.537046 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.537047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.537407 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.537408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.537489 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.537952 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.537961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.538219 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.539661 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.539662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.539662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.539833 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.540042 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.540043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.540043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.540107 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.540397 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.540398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.540444 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.540533 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.540654 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260927 07:04:09.540935 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 23450
I20260927 07:04:09.541040 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:09.541241 21513 external_mini_cluster.cc:1428] Setting key f3813ab99485f603381671eb8e90d9ab
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:09.564105 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40983
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:09.643893 23590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:09.644079 23590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:09.644104 23590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:09.645243 23590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:09.645290 23590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:09.645310 23590 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:04:09.645321 23590 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:04:09.645339 23590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:09.646783 23590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:40983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:09.646967 23590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:09.647141 23590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:09.652016 23596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.652035 23598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:09.652017 23595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:09.652300 23590 server_base.cc:1034] running on GCE node
I20260927 07:04:09.652501 23590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:09.652733 23590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:09.653951 23590 hybrid_clock.cc:648] HybridClock initialized: now 1790492649653943 us; error 43 us; skew 500 ppm
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:09.656574 23590 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:09.657029 23590 webserver.cc:460] Webserver started at http://127.21.2.67:35003/ using document root <none> and password file <none>
I20260927 07:04:09.657279 23590 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:09.657356 23590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:09.657505 23590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:09.658586 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "ea424a0dbe09427b86af0dc7aab75e34"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "45827e3cc6b3c3a042583fe753200c21"
server_key_iv: "df0f6b50dcc9de5ba968a26691d9ec6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.658941 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "ea424a0dbe09427b86af0dc7aab75e34"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "45827e3cc6b3c3a042583fe753200c21"
server_key_iv: "df0f6b50dcc9de5ba968a26691d9ec6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.660377 23590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:09.661118 23605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.661296 23590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:09.661405 23590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "ea424a0dbe09427b86af0dc7aab75e34"
format_stamp: "Formatted at 2026-09-27 07:04:09 on dist-test-slave-ltqz"
server_key: "45827e3cc6b3c3a042583fe753200c21"
server_key_iv: "df0f6b50dcc9de5ba968a26691d9ec6d"
server_key_version: "encryptionkey@0"
I20260927 07:04:09.661477 23590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:09.670060 23590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:09.672385 23590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:09.672542 23590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:09.673094 23590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:09.673146 23590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.673177 23590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:09.673197 23590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.680425 23590 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:45483
I20260927 07:04:09.680779 23590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:09.680786 23720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:45483 every 8 connection(s)
I20260927 07:04:09.683291 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.673515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:45135 (local address 127.21.2.126:40983)
0927 07:04:09.673571 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.673573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.673782 (+ 209us) server_negotiation.cc:407] Connection header received
0927 07:04:09.674313 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.674314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.674326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.674386 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.675029 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.675707 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.676819 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.677026 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.677105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.680202 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.680203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.680204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.680212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.681776 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.682195 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.682196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.682197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.682268 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.682563 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.682564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.682565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.682686 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.682760 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.683114 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.683202 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:04:09.683446 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.673492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.67:45135)
0927 07:04:09.673656 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.673695 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.674197 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.674459 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.674463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.674748 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.674934 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.674939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.675808 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.675809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.676718 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.676721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.677112 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.677113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.677241 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.677783 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.677793 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.680089 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.681885 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.681891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.681894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.682111 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.682350 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.682351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.682352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.682462 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.682751 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.682757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.682940 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.683055 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.683212 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260927 07:04:09.683941 23721 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40983
I20260927 07:04:09.684042 23721 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:09.684207 23721 heartbeater.cc:511] Master 127.21.2.126:40983 requested a full tablet report, sending...
I20260927 07:04:09.684582 23237 ts_manager.cc:194] Registered new tserver with Master: ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:09.684975 23237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:45135
I20260927 07:04:09.688632 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 23590
I20260927 07:04:09.688745 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:09.688969 21513 external_mini_cluster.cc:1428] Setting key 6fa85416ec99e98a687215cd790a260b
I20260927 07:04:09.689839 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:04:09.690804 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.685279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:44945 (local address 127.21.2.126:40983)
0927 07:04:09.685324 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.685326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.685806 (+ 480us) server_negotiation.cc:407] Connection header received
0927 07:04:09.685834 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.685834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.685842 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.685887 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.686130 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.686574 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.687201 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.687422 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.687481 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.688295 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.688296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.688296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.688301 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.689820 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.690139 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.690140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.690140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.690194 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.690402 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.690403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.690403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.690487 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.690531 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.690694 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.690744 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:09.690946 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.685222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.21.2.67:44945)
0927 07:04:09.685737 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.685746 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.685800 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.685889 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.685889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.686033 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.686078 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.686079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.686649 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.686650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.687137 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.687138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.687487 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.687488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.687543 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.687968 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.687973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.688229 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.689876 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.689877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.689877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.690067 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.690252 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.690252 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.690253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.690319 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.690549 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.690550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.690597 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.690682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.690790 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":77,"threads_started":1}
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.712756 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.705445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.21.2.126:40983)
0927 07:04:09.705507 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.705509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.705519 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:09.705583 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.705584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.705601 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.705664 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.706300 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.706788 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.707656 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.707847 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.707922 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.710211 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.710213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.710213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.710221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.711614 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.711976 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.711977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.711978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.712037 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.712231 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.712232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.712233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.712350 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.712451 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.712571 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.712650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:04:09.714322 23237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:09.715906 23237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:09.722216 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41993 (local address 127.0.0.1:53054)
0927 07:04:09.718386 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.718430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.718495 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.718671 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.718674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.718676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:09.718740 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.718741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.719954 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.719955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.720856 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.720859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.721886 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.721887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.721969 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.721982 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.722064 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.722125 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260927 07:04:09.722234 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.21.2.65:41993)
0927 07:04:09.718514 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.718519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.718529 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:09.718574 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.718576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.718590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.718667 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:09.718796 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.719863 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.720955 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.721725 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.721795 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.721850 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.722094 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.722136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260927 07:04:09.722316 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.0.0.1:55742)
0927 07:04:09.718554 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.718566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.718617 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.718973 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.718974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.718976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:09.719032 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.719033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.720133 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.720134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.721230 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.721232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.722102 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.722102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.722162 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.722165 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.722205 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.722242 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":75,"threads_started":1}
I20260927 07:04:09.722823 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.21.2.66:37267)
0927 07:04:09.718821 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.718824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.718832 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:04:09.718863 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.718864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.718876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.718961 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:09.719114 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.720048 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.721318 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.722012 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.722609 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.722671 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.722695 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.722729 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":86,"threads_started":1}
I20260927 07:04:09.722855 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45483 (local address 127.0.0.1:35602)
0927 07:04:09.719045 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.719055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.719138 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.719418 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.719419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.719421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:09.719467 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.719468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.720696 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.720697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.721713 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.721715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.722611 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.722613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.722675 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.722679 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.722734 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.722784 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":53,"threads_started":1}
I20260927 07:04:09.723054 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.718605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.21.2.67:45483)
0927 07:04:09.718744 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.718748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.719111 (+ 363us) server_negotiation.cc:407] Connection header received
0927 07:04:09.719244 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.719247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.719262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.719310 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:09.719545 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.720604 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.721800 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.722462 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.722527 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.722578 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.722767 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.722803 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":101,"threads_started":1}
I20260927 07:04:09.723035 23371 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 (DEFAULT_TABLE table=test-table [id=76116f7d347b418c88755a8b54ece345]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:09.723460 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c02b7c864443f8d42092dc5f12442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.723601 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 (DEFAULT_TABLE table=test-table [id=76116f7d347b418c88755a8b54ece345]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:09.723953 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c02b7c864443f8d42092dc5f12442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.726399 23744 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Bootstrap starting.
I20260927 07:04:09.726552 23745 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Bootstrap starting.
I20260927 07:04:09.727037 23744 tablet_bootstrap.cc:654] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.727046 23745 tablet_bootstrap.cc:654] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.727012 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 (DEFAULT_TABLE table=test-table [id=76116f7d347b418c88755a8b54ece345]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:09.727332 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c02b7c864443f8d42092dc5f12442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.727393 23744 log.cc:824] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:09.727357 23745 log.cc:824] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:09.728164 23745 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: No bootstrap required, opened a new log
I20260927 07:04:09.728164 23744 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: No bootstrap required, opened a new log
I20260927 07:04:09.728232 23745 ts_tablet_manager.cc:1397] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:09.728235 23744 ts_tablet_manager.cc:1397] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:09.729535 23745 raft_consensus.cc:348] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.729647 23745 raft_consensus.cc:374] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.729707 23745 raft_consensus.cc:729] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.729832 23745 consensus_queue.cc:260] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.730101 23745 ts_tablet_manager.cc:1428] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:09.730110 23581 heartbeater.cc:503] Master 127.21.2.126:40983 was elected leader, sending a full tablet report...
I20260927 07:04:09.730211 23744 raft_consensus.cc:348] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.730360 23744 raft_consensus.cc:374] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.730432 23744 raft_consensus.cc:729] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.730544 23744 consensus_queue.cc:260] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.730737 23749 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Bootstrap starting.
I20260927 07:04:09.730803 23744 ts_tablet_manager.cc:1428] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:09.730898 23441 heartbeater.cc:503] Master 127.21.2.126:40983 was elected leader, sending a full tablet report...
I20260927 07:04:09.732012 23749 tablet_bootstrap.cc:654] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.732318 23749 log.cc:824] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:09.732968 23749 tablet_bootstrap.cc:492] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: No bootstrap required, opened a new log
I20260927 07:04:09.733047 23749 ts_tablet_manager.cc:1397] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:09.734583 23749 raft_consensus.cc:348] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.734745 23749 raft_consensus.cc:374] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.734792 23749 raft_consensus.cc:729] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.734929 23749 consensus_queue.cc:260] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.735183 23749 ts_tablet_manager.cc:1428] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:04:09.735239 23721 heartbeater.cc:503] Master 127.21.2.126:40983 was elected leader, sending a full tablet report...
W20260927 07:04:09.781360 23582 tablet.cc:2369] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:09.846935 23748 raft_consensus.cc:482] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:09.847111 23748 raft_consensus.cc:504] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.847522 23748 leader_election.cc:288] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:09.850405 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.847725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45483 (local address 127.21.2.66:44063)
0927 07:04:09.847821 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.847834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.847914 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.848068 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.848069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.848070 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:09.848141 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.848143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.848802 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.848803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.849578 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.849580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.850118 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.850119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.850200 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.850203 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.850275 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.850320 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 07:04:09.850404 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.847796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:44063 (local address 127.21.2.67:45483)
0927 07:04:09.847855 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.847857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.847929 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:04:09.847981 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.847981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.847992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.848065 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:09.848224 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.848706 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.849675 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.850037 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.850121 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.850131 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.850295 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.850328 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:04:09.850540 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.847856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:57415 (local address 127.21.2.65:41993)
0927 07:04:09.847904 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.847906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.848052 (+ 146us) server_negotiation.cc:407] Connection header received
0927 07:04:09.848125 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.848126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.848137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.848176 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:09.848343 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.848915 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.849817 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.850178 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.850266 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.850276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.850438 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.850473 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:09.850564 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.847725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41993 (local address 127.21.2.66:57415)
0927 07:04:09.847961 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.847978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.848059 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.848188 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.848188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.848190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:09.848271 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.848273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.849012 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.849013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.849721 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.849722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.850267 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.850268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.850353 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.850356 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.850433 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.850470 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260927 07:04:09.850730 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c02b7c864443f8d42092dc5f12442" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34" is_pre_election: true
I20260927 07:04:09.850848 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c02b7c864443f8d42092dc5f12442" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce" is_pre_election: true
I20260927 07:04:09.850919 23675 raft_consensus.cc:2454] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 0.
I20260927 07:04:09.850988 23395 raft_consensus.cc:2454] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 0.
I20260927 07:04:09.851186 23469 leader_election.cc:302] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, 88b20e3c35864595bf37106859fc85ad; no voters:
I20260927 07:04:09.851346 23748 raft_consensus.cc:2781] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:09.851428 23748 raft_consensus.cc:482] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:09.851462 23748 raft_consensus.cc:3037] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:09.852339 23748 raft_consensus.cc:504] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.852496 23748 leader_election.cc:288] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 election: Requested vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:09.852643 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c02b7c864443f8d42092dc5f12442" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34"
I20260927 07:04:09.852653 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c02b7c864443f8d42092dc5f12442" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce"
I20260927 07:04:09.852728 23675 raft_consensus.cc:3037] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:09.852728 23395 raft_consensus.cc:3037] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:09.853579 23675 raft_consensus.cc:2454] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 1.
I20260927 07:04:09.853579 23395 raft_consensus.cc:2454] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 1.
I20260927 07:04:09.853754 23469 leader_election.cc:302] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, 88b20e3c35864595bf37106859fc85ad; no voters:
I20260927 07:04:09.853847 23748 raft_consensus.cc:2781] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:09.854002 23748 raft_consensus.cc:686] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 1 LEADER]: Becoming Leader. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Running, Role: LEADER
I20260927 07:04:09.854099 23748 consensus_queue.cc:237] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.855000 23238 catalog_manager.cc:5495] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b20e3c35864595bf37106859fc85ad (127.21.2.66). New cstate: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } health_report { overall_health: UNKNOWN } } }
W20260927 07:04:09.876040 23442 tablet.cc:2369] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:04:09 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492649, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:09.891340 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.884141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40983 (local address 127.0.0.1:48100)
0927 07:04:09.884331 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:09.884341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:09.884407 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:09.884540 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:09.884541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:09.884831 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:09.884921 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.884924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.885588 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.885588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.886110 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:09.886112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.886451 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.886452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:09.886506 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.887228 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:09.887232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:09.888812 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:09.890240 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.890248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.890250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.890469 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.890674 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:09.890675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:09.890675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:09.890739 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:09.891021 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:09.891027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:09.891071 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:04:09.891124 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:04:09.891234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
I20260927 07:04:09.891480 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:09.884193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.21.2.126:40983)
0927 07:04:09.884260 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:09.884263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:09.884406 (+ 143us) server_negotiation.cc:407] Connection header received
0927 07:04:09.884450 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:09.884451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:09.884460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:09.884511 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:09.885012 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.885515 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.886180 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:09.886363 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:09.886416 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:09.888943 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:09.888945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:09.888945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:09.888952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:09.890158 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.890535 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.890535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.890536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.890585 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:09.890821 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:09.890822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:09.890822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:09.890951 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:09.891014 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:09.891158 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:04:09.891280 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":35}
I20260927 07:04:09.893630 23238 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48100:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:04:09.893857 23238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:09.895458 23371 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2d2d080db24f4088073a678e213a3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65027402325d41dd8119dcc6253db8b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:09.895503 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2d2d080db24f4088073a678e213a3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65027402325d41dd8119dcc6253db8b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:09.895514 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2d2d080db24f4088073a678e213a3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65027402325d41dd8119dcc6253db8b9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:04:09.895641 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2d2d080db24f4088073a678e213a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.895641 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2d2d080db24f4088073a678e213a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.895690 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2d2d080db24f4088073a678e213a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:09.896960 23749 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: Bootstrap starting.
I20260927 07:04:09.897079 23744 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: Bootstrap starting.
I20260927 07:04:09.897094 23745 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: Bootstrap starting.
I20260927 07:04:09.897505 23744 tablet_bootstrap.cc:654] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.897517 23745 tablet_bootstrap.cc:654] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.898090 23744 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: No bootstrap required, opened a new log
I20260927 07:04:09.898144 23744 ts_tablet_manager.cc:1397] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:09.898226 23745 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: No bootstrap required, opened a new log
I20260927 07:04:09.898275 23745 ts_tablet_manager.cc:1397] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:09.898315 23744 raft_consensus.cc:348] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.898391 23744 raft_consensus.cc:374] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.898420 23744 raft_consensus.cc:729] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.898448 23745 raft_consensus.cc:348] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.898484 23744 consensus_queue.cc:260] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.898519 23745 raft_consensus.cc:374] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.898540 23745 raft_consensus.cc:729] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.898577 23744 ts_tablet_manager.cc:1428] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:09.898590 23745 consensus_queue.cc:260] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.898687 23745 ts_tablet_manager.cc:1428] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.898743 23748 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:09.898814 23748 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:09.898808 23750 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:09.898883 23750 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:09.899243 23749 tablet_bootstrap.cc:654] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:09.900045 23749 tablet_bootstrap.cc:492] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: No bootstrap required, opened a new log
I20260927 07:04:09.900108 23749 ts_tablet_manager.cc:1397] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 07:04:09.900266 23749 raft_consensus.cc:348] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.900343 23749 raft_consensus.cc:374] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:09.900377 23749 raft_consensus.cc:729] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Initialized, Role: FOLLOWER
I20260927 07:04:09.900429 23749 consensus_queue.cc:260] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:09.900519 23749 ts_tablet_manager.cc:1428] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:09.900591 23752 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:09.900708 23752 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 07:04:09.931591 23722 tablet.cc:2369] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:10.048303 23748 raft_consensus.cc:482] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:10.048395 23748 raft_consensus.cc:504] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.048542 23748 leader_election.cc:288] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:10.048770 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2d2d080db24f4088073a678e213a3b" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34" is_pre_election: true
I20260927 07:04:10.048805 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2d2d080db24f4088073a678e213a3b" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce" is_pre_election: true
I20260927 07:04:10.048890 23675 raft_consensus.cc:2454] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 0.
I20260927 07:04:10.048918 23395 raft_consensus.cc:2454] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 0.
I20260927 07:04:10.049074 23466 leader_election.cc:302] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b20e3c35864595bf37106859fc85ad, ea424a0dbe09427b86af0dc7aab75e34; no voters:
I20260927 07:04:10.049185 23748 raft_consensus.cc:2781] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:10.049237 23748 raft_consensus.cc:482] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:10.049268 23748 raft_consensus.cc:3037] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.049952 23748 raft_consensus.cc:504] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.050107 23748 leader_election.cc:288] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 election: Requested vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:10.050235 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2d2d080db24f4088073a678e213a3b" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce"
I20260927 07:04:10.050307 23395 raft_consensus.cc:3037] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.050294 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2d2d080db24f4088073a678e213a3b" candidate_uuid: "88b20e3c35864595bf37106859fc85ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34"
I20260927 07:04:10.050366 23675 raft_consensus.cc:3037] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.050943 23395 raft_consensus.cc:2454] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 1.
I20260927 07:04:10.050967 23675 raft_consensus.cc:2454] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b20e3c35864595bf37106859fc85ad in term 1.
I20260927 07:04:10.051112 23469 leader_election.cc:302] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, 88b20e3c35864595bf37106859fc85ad; no voters:
I20260927 07:04:10.051200 23748 raft_consensus.cc:2781] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:10.051249 23748 raft_consensus.cc:686] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [term 1 LEADER]: Becoming Leader. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Running, Role: LEADER
I20260927 07:04:10.051298 23748 consensus_queue.cc:237] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.051635 23754 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: TxnStatusTablet state changed. Reason: New leader 88b20e3c35864595bf37106859fc85ad. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.051724 23754 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:10.051887 23766 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:04:10.052030 23766 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:04:10.052033 23238 catalog_manager.cc:5495] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b20e3c35864595bf37106859fc85ad (127.21.2.66). New cstate: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:10.081131 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.078136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.0.0.1:55744)
0927 07:04:10.078214 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.078228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.078283 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.078508 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.078509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.078511 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:10.078563 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.078565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.079508 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.079509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.080185 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.080187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.080661 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.080662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.080709 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.080723 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.080998 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.081005 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.081033 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.081070 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 07:04:10.081144 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.078231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.21.2.66:37267)
0927 07:04:10.078282 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.078284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.078294 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:10.078377 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.078378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.078399 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.078485 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.078665 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.079429 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.080271 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.080585 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.080656 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.080813 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.080936 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.081015 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.081039 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.081059 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:10.083079 23675 raft_consensus.cc:1261] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Refusing update from remote peer 88b20e3c35864595bf37106859fc85ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.083087 23395 raft_consensus.cc:1261] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Refusing update from remote peer 88b20e3c35864595bf37106859fc85ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.083493 23748 consensus_queue.cc:1035] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.083667 23754 consensus_queue.cc:1035] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.084956 23752 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: TxnStatusTablet state changed. Reason: New leader 88b20e3c35864595bf37106859fc85ad. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.085072 23752 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:10.085017 23750 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: TxnStatusTablet state changed. Reason: New leader 88b20e3c35864595bf37106859fc85ad. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.085103 23750 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:10.086004 23748 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.086082 23748 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:10.086123 23748 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.086164 23748 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 88b20e3c35864595bf37106859fc85ad: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:04:10.086422 23752 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.086491 23752 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P ea424a0dbe09427b86af0dc7aab75e34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:10.086596 23750 tablet_replica.cc:439] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88b20e3c35864595bf37106859fc85ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } } }
I20260927 07:04:10.086665 23750 tablet_replica.cc:442] T 8d2d2d080db24f4088073a678e213a3b P 4c2d52f6204547d2bf5a231e76566cce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:04:10.087006 23770 mvcc.cc:205] Tried to move back new op lower bound from 7333857894737620992 to 7333857894610657280. Current Snapshot: MvccSnapshot[applied={T|T < 7333857894737620992}]
I20260927 07:04:10.087239 23771 mvcc.cc:205] Tried to move back new op lower bound from 7333857894737620992 to 7333857894610657280. Current Snapshot: MvccSnapshot[applied={T|T < 7333857894737620992}]
I20260927 07:04:10.090341 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.088134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.21.2.66:37267)
0927 07:04:10.088179 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.088180 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.088188 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:04:10.088233 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.088234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.088257 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.088316 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.088449 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.088902 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.089649 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.089854 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.089929 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.090080 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.090112 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.090181 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.090234 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.090257 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:10.095435 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.092864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:45987 (local address 127.21.2.66:37267)
0927 07:04:10.092906 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.092908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.093041 (+ 133us) server_negotiation.cc:407] Connection header received
0927 07:04:10.093100 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.093100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.093109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.093162 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.093540 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.093978 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.094715 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.094912 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.094994 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.095170 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.095200 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.095252 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.095311 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.095325 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:04:10.095448 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.092796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.21.2.66:45987)
0927 07:04:10.092965 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.092975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.093051 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.093164 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.093165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.093166 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:10.093466 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.093467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.094055 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.094056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.094630 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.094632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.094996 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.094996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.095069 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.095081 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.095258 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.095260 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.095299 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.095340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260927 07:04:10.097478 23675 raft_consensus.cc:1261] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Refusing update from remote peer 88b20e3c35864595bf37106859fc85ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.097527 23395 raft_consensus.cc:1261] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Refusing update from remote peer 88b20e3c35864595bf37106859fc85ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.097703 23748 consensus_queue.cc:1035] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.097839 23748 consensus_queue.cc:1035] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.099144 23769 mvcc.cc:205] Tried to move back new op lower bound from 7333857894798057472 to 7333857893803032576. Current Snapshot: MvccSnapshot[applied={T|T < 7333857894798057472}]
I20260927 07:04:10.115346 23238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:10.115504 23238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:10.116297 23238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Sending DeleteTablet for 3 replicas of tablet 3d2c02b7c864443f8d42092dc5f12442
I20260927 07:04:10.116601 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:10 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:53054
I20260927 07:04:10.116840 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:10 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:35602
I20260927 07:04:10.117033 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c02b7c864443f8d42092dc5f12442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:10 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55742
I20260927 07:04:10.117096 23785 tablet_replica.cc:330] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: stopping tablet replica
I20260927 07:04:10.117075 23786 tablet_replica.cc:330] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: stopping tablet replica
I20260927 07:04:10.117208 23785 raft_consensus.cc:2229] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:10.117274 23785 raft_consensus.cc:2258] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:10.117353 23786 raft_consensus.cc:2229] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:10.117475 23786 raft_consensus.cc:2258] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:10.117511 23787 tablet_replica.cc:330] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: stopping tablet replica
I20260927 07:04:10.117491 23238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 07:04:10.117615 23787 raft_consensus.cc:2229] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 1 LEADER]: Raft consensus shutting down.
W20260927 07:04:10.117650 23238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:10.117703 23787 raft_consensus.cc:2258] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:10.118109 23787 ts_tablet_manager.cc:1905] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:10.118296 23786 ts_tablet_manager.cc:1905] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:10.119081 23515 tablet_service.cc:1467] Processing CreateTablet for tablet cc1cc5578260470987b16f251ec78565 (DEFAULT_TABLE table=test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:10.119208 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc1cc5578260470987b16f251ec78565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:10.119784 23787 ts_tablet_manager.cc:1918] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:10.119861 23787 log.cc:1196] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/3d2c02b7c864443f8d42092dc5f12442
I20260927 07:04:10.120100 23787 ts_tablet_manager.cc:1939] T 3d2c02b7c864443f8d42092dc5f12442 P 88b20e3c35864595bf37106859fc85ad: Deleting consensus metadata
I20260927 07:04:10.120507 23222 catalog_manager.cc:4841] TS 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267): tablet 3d2c02b7c864443f8d42092dc5f12442 (table test-table [id=76116f7d347b418c88755a8b54ece345]) successfully deleted
I20260927 07:04:10.120534 23786 ts_tablet_manager.cc:1918] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:10.120680 23786 log.cc:1196] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/3d2c02b7c864443f8d42092dc5f12442
I20260927 07:04:10.120735 23655 tablet_service.cc:1467] Processing CreateTablet for tablet cc1cc5578260470987b16f251ec78565 (DEFAULT_TABLE table=test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:10.120839 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc1cc5578260470987b16f251ec78565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:10.120965 23786 ts_tablet_manager.cc:1939] T 3d2c02b7c864443f8d42092dc5f12442 P ea424a0dbe09427b86af0dc7aab75e34: Deleting consensus metadata
I20260927 07:04:10.121379 23222 catalog_manager.cc:4841] TS ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483): tablet 3d2c02b7c864443f8d42092dc5f12442 (table test-table [id=76116f7d347b418c88755a8b54ece345]) successfully deleted
I20260927 07:04:10.121832 23749 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Bootstrap starting.
I20260927 07:04:10.121861 23745 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Bootstrap starting.
I20260927 07:04:10.122153 23745 tablet_bootstrap.cc:654] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:10.122201 23749 tablet_bootstrap.cc:654] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:10.122753 23749 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: No bootstrap required, opened a new log
I20260927 07:04:10.122756 23371 tablet_service.cc:1467] Processing CreateTablet for tablet cc1cc5578260470987b16f251ec78565 (DEFAULT_TABLE table=test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:10.122813 23749 ts_tablet_manager.cc:1397] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:10.122866 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc1cc5578260470987b16f251ec78565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:10.122963 23749 raft_consensus.cc:348] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.123039 23749 raft_consensus.cc:374] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:10.123061 23749 raft_consensus.cc:729] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Initialized, Role: FOLLOWER
I20260927 07:04:10.123081 23745 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: No bootstrap required, opened a new log
I20260927 07:04:10.123117 23745 ts_tablet_manager.cc:1397] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:10.123104 23749 consensus_queue.cc:260] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.123200 23749 ts_tablet_manager.cc:1428] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:10.123243 23745 raft_consensus.cc:348] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.123314 23745 raft_consensus.cc:374] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:10.123339 23745 raft_consensus.cc:729] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:10.123401 23745 consensus_queue.cc:260] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.123508 23745 ts_tablet_manager.cc:1428] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:10.123771 23785 ts_tablet_manager.cc:1905] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:10.124809 23744 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Bootstrap starting.
I20260927 07:04:10.125200 23744 tablet_bootstrap.cc:654] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:10.125427 23785 ts_tablet_manager.cc:1918] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:10.125485 23785 log.cc:1196] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/3d2c02b7c864443f8d42092dc5f12442
I20260927 07:04:10.125762 23785 ts_tablet_manager.cc:1939] T 3d2c02b7c864443f8d42092dc5f12442 P 4c2d52f6204547d2bf5a231e76566cce: Deleting consensus metadata
I20260927 07:04:10.125836 23744 tablet_bootstrap.cc:492] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: No bootstrap required, opened a new log
I20260927 07:04:10.125875 23744 ts_tablet_manager.cc:1397] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:10.126021 23744 raft_consensus.cc:348] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.126096 23744 raft_consensus.cc:374] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:10.126120 23744 raft_consensus.cc:729] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Initialized, Role: FOLLOWER
I20260927 07:04:10.126183 23744 consensus_queue.cc:260] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.126286 23744 ts_tablet_manager.cc:1428] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:10.126597 23225 catalog_manager.cc:4841] TS 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993): tablet 3d2c02b7c864443f8d42092dc5f12442 (table test-table [id=76116f7d347b418c88755a8b54ece345]) successfully deleted
I20260927 07:04:10.171620 23750 raft_consensus.cc:482] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:10.171747 23750 raft_consensus.cc:504] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.172071 23750 leader_election.cc:288] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:10.176235 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.172346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:39157 (local address 127.21.2.66:37267)
0927 07:04:10.172418 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.172420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.172482 (+ 62us) server_negotiation.cc:407] Connection header received
0927 07:04:10.172605 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.172606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.172618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.172649 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:10.172945 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.173677 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175279 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.175838 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175885 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.175897 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.176130 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.176166 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:04:10.176234 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.172527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:35009 (local address 127.21.2.67:45483)
0927 07:04:10.172572 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.172574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.172610 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:04:10.172677 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.172678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.172687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.172772 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:10.172946 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.173675 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175280 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.175843 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175885 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.175895 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.176129 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.176167 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:10.176489 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.172425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45483 (local address 127.21.2.65:35009)
0927 07:04:10.172560 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.172569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.172619 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.172796 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.172796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.172798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:10.172853 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.172854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.173779 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.173779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.175170 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.175173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.175962 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.176040 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.176046 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.176337 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.176389 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":59,"thread_start_us":77,"threads_started":1}
I20260927 07:04:10.176492 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.172240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.21.2.65:39157)
0927 07:04:10.172415 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.172426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.172516 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.172721 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.172722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.172730 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:10.172826 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.172829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.173806 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.173807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.175171 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.175173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.175961 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.175962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.176040 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.176046 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.176374 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.176404 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":72,"thread_start_us":84,"threads_started":1}
I20260927 07:04:10.176815 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc1cc5578260470987b16f251ec78565" candidate_uuid: "4c2d52f6204547d2bf5a231e76566cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34" is_pre_election: true
I20260927 07:04:10.176856 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc1cc5578260470987b16f251ec78565" candidate_uuid: "4c2d52f6204547d2bf5a231e76566cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad" is_pre_election: true
I20260927 07:04:10.176920 23675 raft_consensus.cc:2454] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2d52f6204547d2bf5a231e76566cce in term 0.
I20260927 07:04:10.176967 23535 raft_consensus.cc:2454] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c2d52f6204547d2bf5a231e76566cce in term 0.
I20260927 07:04:10.177150 23326 leader_election.cc:302] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, 88b20e3c35864595bf37106859fc85ad; no voters:
I20260927 07:04:10.177274 23750 raft_consensus.cc:2781] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:10.177317 23750 raft_consensus.cc:482] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:10.177353 23750 raft_consensus.cc:3037] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.178018 23750 raft_consensus.cc:504] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.178153 23750 leader_election.cc:288] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [CANDIDATE]: Term 1 election: Requested vote from peers 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267), ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483)
I20260927 07:04:10.178309 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc1cc5578260470987b16f251ec78565" candidate_uuid: "4c2d52f6204547d2bf5a231e76566cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad"
I20260927 07:04:10.178332 23675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc1cc5578260470987b16f251ec78565" candidate_uuid: "4c2d52f6204547d2bf5a231e76566cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea424a0dbe09427b86af0dc7aab75e34"
I20260927 07:04:10.178406 23535 raft_consensus.cc:3037] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.178416 23675 raft_consensus.cc:3037] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:10.179159 23535 raft_consensus.cc:2454] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2d52f6204547d2bf5a231e76566cce in term 1.
I20260927 07:04:10.179181 23675 raft_consensus.cc:2454] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c2d52f6204547d2bf5a231e76566cce in term 1.
I20260927 07:04:10.179333 23326 leader_election.cc:302] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, ea424a0dbe09427b86af0dc7aab75e34; no voters:
I20260927 07:04:10.179498 23750 raft_consensus.cc:2781] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:10.179631 23750 raft_consensus.cc:686] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 1 LEADER]: Becoming Leader. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Running, Role: LEADER
I20260927 07:04:10.179710 23750 consensus_queue.cc:237] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:10.180474 23237 catalog_manager.cc:5495] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65). New cstate: current_term: 1 leader_uuid: "4c2d52f6204547d2bf5a231e76566cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:10.187269 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.184807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.21.2.65:41993)
0927 07:04:10.184844 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.184846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.184903 (+ 57us) server_negotiation.cc:407] Connection header received
0927 07:04:10.184976 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.184976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.184990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.185039 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.185144 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.185589 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.186604 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.186774 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.186825 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.186978 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.187067 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.187119 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.187179 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.187196 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:10.192318 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.189629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54873 (local address 127.21.2.66:37267)
0927 07:04:10.189678 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.189680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.189740 (+ 60us) server_negotiation.cc:407] Connection header received
0927 07:04:10.189811 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.189812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.189822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.189861 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.189987 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.190434 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.191626 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.191828 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.191892 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.192092 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.192120 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.192159 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.192246 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.192263 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:10.192561 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.189540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.21.2.65:54873)
0927 07:04:10.189685 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.189697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.189756 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.189875 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.189876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.189877 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:10.189937 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.189938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.190503 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.190504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.191503 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.191506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.191923 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.191924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.191993 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.192011 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.192187 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.192189 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.192242 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.192285 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 07:04:10.194804 23675 raft_consensus.cc:1261] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Refusing update from remote peer 4c2d52f6204547d2bf5a231e76566cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.194832 23535 raft_consensus.cc:1261] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Refusing update from remote peer 4c2d52f6204547d2bf5a231e76566cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:10.195081 23750 consensus_queue.cc:1035] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.195237 23793 consensus_queue.cc:1035] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:10.202402 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.199787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41993 (local address 127.21.2.66:44591)
0927 07:04:10.199829 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:10.199840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:10.199879 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:10.200105 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:10.200106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:10.200108 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:10.200149 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.200150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.200748 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.200749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.201557 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:10.201559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.202052 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.202052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:10.202097 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.202103 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.202281 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.202282 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:04:10.202317 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 07:04:10.202352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 07:04:10.202407 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:10.199953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:44591 (local address 127.21.2.65:41993)
0927 07:04:10.199992 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:10.199994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:10.200001 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:04:10.200021 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:10.200022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:10.200032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:10.200088 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:10.200206 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.200673 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.201670 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:10.201964 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:10.202044 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:10.202181 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:10.202215 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:10.202267 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:10.202321 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:04:10.202347 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:04:10.202772 23771 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 07:04:11.728137 23237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:11.728256 23237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:11.729123 23237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Sending DeleteTablet for 3 replicas of tablet cc1cc5578260470987b16f251ec78565
I20260927 07:04:11.729374 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet cc1cc5578260470987b16f251ec78565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:11 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:53054
I20260927 07:04:11.729434 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet cc1cc5578260470987b16f251ec78565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:11 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55742
I20260927 07:04:11.729605 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet cc1cc5578260470987b16f251ec78565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:11 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:35602
I20260927 07:04:11.729622 23825 tablet_replica.cc:330] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: stopping tablet replica
I20260927 07:04:11.729709 23825 raft_consensus.cc:2229] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:11.729785 23825 raft_consensus.cc:2258] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:11.729817 23826 tablet_replica.cc:330] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: stopping tablet replica
I20260927 07:04:11.729903 23826 raft_consensus.cc:2229] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:11.729969 23826 raft_consensus.cc:2258] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:11.730126 23825 ts_tablet_manager.cc:1905] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:11.730185 23826 ts_tablet_manager.cc:1905] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:11.731951 23826 ts_tablet_manager.cc:1918] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:04:11.732079 23826 log.cc:1196] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/cc1cc5578260470987b16f251ec78565
I20260927 07:04:11.732127 23825 ts_tablet_manager.cc:1918] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:04:11.732235 23825 log.cc:1196] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/cc1cc5578260470987b16f251ec78565
I20260927 07:04:11.732342 23826 ts_tablet_manager.cc:1939] T cc1cc5578260470987b16f251ec78565 P ea424a0dbe09427b86af0dc7aab75e34: Deleting consensus metadata
I20260927 07:04:11.732517 23825 ts_tablet_manager.cc:1939] T cc1cc5578260470987b16f251ec78565 P 88b20e3c35864595bf37106859fc85ad: Deleting consensus metadata
I20260927 07:04:11.732688 23222 catalog_manager.cc:4841] TS ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483): tablet cc1cc5578260470987b16f251ec78565 (table test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]) successfully deleted
I20260927 07:04:11.732817 23222 catalog_manager.cc:4841] TS 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267): tablet cc1cc5578260470987b16f251ec78565 (table test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]) successfully deleted
I20260927 07:04:11.734792 23824 tablet_replica.cc:330] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: stopping tablet replica
I20260927 07:04:11.734845 23824 raft_consensus.cc:2229] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:11.734957 23824 raft_consensus.cc:2258] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:11.735183 23824 ts_tablet_manager.cc:1905] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:11.736732 23824 ts_tablet_manager.cc:1918] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:04:11.736797 23824 log.cc:1196] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/cc1cc5578260470987b16f251ec78565
I20260927 07:04:11.737048 23824 ts_tablet_manager.cc:1939] T cc1cc5578260470987b16f251ec78565 P 4c2d52f6204547d2bf5a231e76566cce: Deleting consensus metadata
I20260927 07:04:11.737350 23225 catalog_manager.cc:4841] TS 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993): tablet cc1cc5578260470987b16f251ec78565 (table test-table [id=ccff1f6aa311490b9b1ca880cb7ff700]) successfully deleted
Sep 27 07:04:11 dist-test-slave-ltqz krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:04:11 dist-test-slave-ltqz krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:11.753468 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:11.746390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.21.2.126:40983)
0927 07:04:11.746530 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:11.746532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:11.746623 (+ 91us) server_negotiation.cc:407] Connection header received
0927 07:04:11.746680 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:11.746681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:11.746691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:11.746753 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:11.747321 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:11.747874 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:11.748593 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:11.748777 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:11.748826 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:11.751056 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:11.751057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:11.751058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:11.751065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:11.752407 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:11.752779 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:11.752781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:11.752781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:11.752852 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:11.753039 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:11.753040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:11.753040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:11.753149 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:11.753274 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:11.753330 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 07:04:11.753397 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260927 07:04:11.754911 23237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:11.755069 23237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:11.756531 23371 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01ca665e654928817f82608e0ed55c (DEFAULT_TABLE table=test-table [id=ac9f3975414c4382be802e6d42acea72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:11.756590 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01ca665e654928817f82608e0ed55c (DEFAULT_TABLE table=test-table [id=ac9f3975414c4382be802e6d42acea72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:11.756611 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01ca665e654928817f82608e0ed55c (DEFAULT_TABLE table=test-table [id=ac9f3975414c4382be802e6d42acea72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:11.756673 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01ca665e654928817f82608e0ed55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:11.756697 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01ca665e654928817f82608e0ed55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:11.756723 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01ca665e654928817f82608e0ed55c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:11.757701 23835 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Bootstrap starting.
I20260927 07:04:11.758080 23836 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Bootstrap starting.
I20260927 07:04:11.758097 23835 tablet_bootstrap.cc:654] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:11.758219 23837 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Bootstrap starting.
I20260927 07:04:11.758441 23836 tablet_bootstrap.cc:654] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:11.758540 23837 tablet_bootstrap.cc:654] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:11.758780 23835 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: No bootstrap required, opened a new log
I20260927 07:04:11.758863 23835 ts_tablet_manager.cc:1397] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:11.759001 23835 raft_consensus.cc:348] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759081 23835 raft_consensus.cc:374] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:11.759080 23836 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: No bootstrap required, opened a new log
I20260927 07:04:11.759119 23835 raft_consensus.cc:729] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Initialized, Role: FOLLOWER
I20260927 07:04:11.759136 23836 ts_tablet_manager.cc:1397] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:11.759186 23837 tablet_bootstrap.cc:492] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: No bootstrap required, opened a new log
I20260927 07:04:11.759176 23835 consensus_queue.cc:260] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759230 23837 ts_tablet_manager.cc:1397] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:11.759243 23836 raft_consensus.cc:348] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759332 23836 raft_consensus.cc:374] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:11.759332 23835 ts_tablet_manager.cc:1428] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:11.759382 23836 raft_consensus.cc:729] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Initialized, Role: FOLLOWER
I20260927 07:04:11.759327 23837 raft_consensus.cc:348] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759431 23836 consensus_queue.cc:260] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759469 23837 raft_consensus.cc:374] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:11.759503 23837 raft_consensus.cc:729] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:11.759533 23836 ts_tablet_manager.cc:1428] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:11.759553 23837 consensus_queue.cc:260] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:11.759697 23837 ts_tablet_manager.cc:1428] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:12.034389 23841 raft_consensus.cc:482] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:12.034509 23841 raft_consensus.cc:504] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.034956 23841 leader_election.cc:288] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267)
I20260927 07:04:12.038067 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.035190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35797 (local address 127.21.2.65:41993)
0927 07:04:12.035340 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:12.035343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:12.035422 (+ 79us) server_negotiation.cc:407] Connection header received
0927 07:04:12.035516 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:12.035517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:12.035527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:12.035620 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:12.035784 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.036271 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037262 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.037659 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037756 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.037769 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:12.037956 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 07:04:12.037999 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260927 07:04:12.038072 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.035132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41993 (local address 127.21.2.67:35797)
0927 07:04:12.035334 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:12.035347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:12.035440 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:12.035624 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:12.035625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:12.035627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:12.035697 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.035699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.036357 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.036358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.037165 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.037167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.037780 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.037868 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.037872 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:12.037935 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 07:04:12.037980 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260927 07:04:12.038213 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.035198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.21.2.67:36331)
0927 07:04:12.035397 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:12.035409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:12.035476 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:12.035646 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:12.035647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:12.035648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:12.035701 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.035703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.036377 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.036378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.037180 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.037182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.037754 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.037828 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.037906 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 07:04:12.038031 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:04:12.038082 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":43,"thread_start_us":110,"threads_started":1}
I20260927 07:04:12.038327 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01ca665e654928817f82608e0ed55c" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce" is_pre_election: true
I20260927 07:04:12.038426 23395 raft_consensus.cc:2454] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 0.
I20260927 07:04:12.038602 23609 leader_election.cc:302] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, ea424a0dbe09427b86af0dc7aab75e34; no voters:
I20260927 07:04:12.038653 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.035246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:36331 (local address 127.21.2.66:37267)
0927 07:04:12.035419 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:12.035421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:12.035459 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:04:12.035540 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:12.035541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:12.035554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:12.035639 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:12.035796 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.036278 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037279 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.037664 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.037752 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.037765 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:12.038046 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 07:04:12.038097 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260927 07:04:12.038758 23841 raft_consensus.cc:2781] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:12.038792 23841 raft_consensus.cc:482] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:12.038821 23841 raft_consensus.cc:3037] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.038965 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01ca665e654928817f82608e0ed55c" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad" is_pre_election: true
I20260927 07:04:12.039062 23535 raft_consensus.cc:2454] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 0.
I20260927 07:04:12.039737 23841 raft_consensus.cc:504] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.039888 23841 leader_election.cc:288] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 election: Requested vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267)
I20260927 07:04:12.040046 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01ca665e654928817f82608e0ed55c" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce"
I20260927 07:04:12.040125 23395 raft_consensus.cc:3037] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.040102 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01ca665e654928817f82608e0ed55c" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad"
I20260927 07:04:12.040216 23535 raft_consensus.cc:3037] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.040769 23395 raft_consensus.cc:2454] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 1.
I20260927 07:04:12.040968 23535 raft_consensus.cc:2454] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 1.
I20260927 07:04:12.041074 23609 leader_election.cc:302] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, ea424a0dbe09427b86af0dc7aab75e34; no voters:
I20260927 07:04:12.041224 23841 raft_consensus.cc:2781] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:12.041429 23841 raft_consensus.cc:686] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 1 LEADER]: Becoming Leader. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Running, Role: LEADER
I20260927 07:04:12.041525 23841 consensus_queue.cc:237] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.049286 23238 catalog_manager.cc:5495] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "ea424a0dbe09427b86af0dc7aab75e34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } health_report { overall_health: HEALTHY } } }
I20260927 07:04:12.116200 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.113501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.21.2.67:45483)
0927 07:04:12.113675 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:12.113677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:12.113683 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:04:12.113718 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:12.113718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:12.113736 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:12.113833 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:12.113983 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.114485 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.115246 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.115549 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.115626 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.115815 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:12.115940 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:12.116027 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:12.116070 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 07:04:12.116092 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260927 07:04:12.120615 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.118394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37267 (local address 127.21.2.67:48427)
0927 07:04:12.118545 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:12.118555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:12.118606 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:12.118710 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:12.118711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:12.118712 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:12.118758 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.118759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.119324 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.119325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.119952 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.119954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.120272 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.120273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.120321 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.120332 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:12.120472 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:12.120474 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:12.120515 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 07:04:12.120548 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260927 07:04:12.120633 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.118547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48427 (local address 127.21.2.66:37267)
0927 07:04:12.118595 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:12.118597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:12.118606 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:12.118651 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:12.118652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:12.118661 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:12.118702 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:12.118820 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.119259 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.120021 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.120199 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.120252 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.120394 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:12.120417 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:12.120456 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:12.120516 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 07:04:12.120533 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:04:12.122534 23395 raft_consensus.cc:1261] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Refusing update from remote peer ea424a0dbe09427b86af0dc7aab75e34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:12.122599 23535 raft_consensus.cc:1261] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Refusing update from remote peer ea424a0dbe09427b86af0dc7aab75e34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:12.122830 23841 consensus_queue.cc:1035] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:12.123001 23846 consensus_queue.cc:1035] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:12.130599 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.128345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:46337 (local address 127.21.2.67:45483)
0927 07:04:12.128384 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:12.128386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:12.128393 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:04:12.128418 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:12.128419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:12.128427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:12.128476 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:12.128637 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.129092 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.129845 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.130078 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.130167 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.130334 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:12.130370 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:12.130417 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:12.130505 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 07:04:12.130527 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:12.130607 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:12.128184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45483 (local address 127.21.2.66:46337)
0927 07:04:12.128301 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:12.128313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:12.128365 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:12.128509 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:12.128511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:12.128513 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:04:12.128557 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.128558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.129181 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.129182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.129748 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:12.129750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:12.130171 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:12.130172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:12.130239 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:12.130244 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:04:12.130432 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:04:12.130434 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:12.130484 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:04:12.130536 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":85,"threads_started":1}
I20260927 07:04:12.136812 23238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:12.136909 23238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:12.137529 23238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Sending DeleteTablet for 3 replicas of tablet 8e01ca665e654928817f82608e0ed55c
I20260927 07:04:12.137826 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01ca665e654928817f82608e0ed55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:12 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:53054
I20260927 07:04:12.137945 23824 tablet_replica.cc:330] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: stopping tablet replica
I20260927 07:04:12.137948 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01ca665e654928817f82608e0ed55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:12 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:35602
I20260927 07:04:12.137996 23824 raft_consensus.cc:2229] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:12.138021 23826 tablet_replica.cc:330] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: stopping tablet replica
I20260927 07:04:12.138041 23824 raft_consensus.cc:2258] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:12.138059 23826 raft_consensus.cc:2229] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:12.138140 23826 raft_consensus.cc:2258] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:12.138176 23238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:12.138320 23238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:12.138435 23824 ts_tablet_manager.cc:1905] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:12.138432 23826 ts_tablet_manager.cc:1905] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:12.139421 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01ca665e654928817f82608e0ed55c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:12 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55742
I20260927 07:04:12.139508 23825 tablet_replica.cc:330] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: stopping tablet replica
I20260927 07:04:12.139552 23825 raft_consensus.cc:2229] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:12.139590 23825 raft_consensus.cc:2258] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:12.139828 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 83d34ebd170e47ba8b2715768236c201 (DEFAULT_TABLE table=test-table [id=b84b60deaa8f4cf7a06bd5e98d33c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:12.139935 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d34ebd170e47ba8b2715768236c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:12.140085 23824 ts_tablet_manager.cc:1918] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:12.140132 23824 log.cc:1196] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/8e01ca665e654928817f82608e0ed55c
I20260927 07:04:12.140336 23824 ts_tablet_manager.cc:1939] T 8e01ca665e654928817f82608e0ed55c P 4c2d52f6204547d2bf5a231e76566cce: Deleting consensus metadata
I20260927 07:04:12.141165 23835 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Bootstrap starting.
I20260927 07:04:12.141211 23371 tablet_service.cc:1467] Processing CreateTablet for tablet 83d34ebd170e47ba8b2715768236c201 (DEFAULT_TABLE table=test-table [id=b84b60deaa8f4cf7a06bd5e98d33c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:12.141312 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d34ebd170e47ba8b2715768236c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:12.141546 23835 tablet_bootstrap.cc:654] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:12.142375 23836 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: Bootstrap starting.
I20260927 07:04:12.142472 23225 catalog_manager.cc:4841] TS 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993): tablet 8e01ca665e654928817f82608e0ed55c (table test-table [id=ac9f3975414c4382be802e6d42acea72]) successfully deleted
I20260927 07:04:12.142500 23835 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: No bootstrap required, opened a new log
I20260927 07:04:12.142547 23835 ts_tablet_manager.cc:1397] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:12.142721 23836 tablet_bootstrap.cc:654] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:12.142705 23835 raft_consensus.cc:348] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.142767 23835 raft_consensus.cc:374] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:12.142788 23835 raft_consensus.cc:729] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Initialized, Role: FOLLOWER
I20260927 07:04:12.142817 23825 ts_tablet_manager.cc:1905] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:12.142830 23835 consensus_queue.cc:260] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.142938 23835 ts_tablet_manager.cc:1428] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:12.144016 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 83d34ebd170e47ba8b2715768236c201 (DEFAULT_TABLE table=test-table [id=b84b60deaa8f4cf7a06bd5e98d33c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:12.144063 23825 ts_tablet_manager.cc:1918] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:12.144104 23825 log.cc:1196] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/8e01ca665e654928817f82608e0ed55c
I20260927 07:04:12.144124 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d34ebd170e47ba8b2715768236c201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:12.144325 23825 ts_tablet_manager.cc:1939] T 8e01ca665e654928817f82608e0ed55c P 88b20e3c35864595bf37106859fc85ad: Deleting consensus metadata
I20260927 07:04:12.144341 23836 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: No bootstrap required, opened a new log
I20260927 07:04:12.144387 23836 ts_tablet_manager.cc:1397] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:04:12.144505 23836 raft_consensus.cc:348] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.144567 23836 raft_consensus.cc:374] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:12.144586 23836 raft_consensus.cc:729] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2d52f6204547d2bf5a231e76566cce, State: Initialized, Role: FOLLOWER
I20260927 07:04:12.144620 23222 catalog_manager.cc:4841] TS 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267): tablet 8e01ca665e654928817f82608e0ed55c (table test-table [id=ac9f3975414c4382be802e6d42acea72]) successfully deleted
I20260927 07:04:12.144630 23836 consensus_queue.cc:260] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.144735 23836 ts_tablet_manager.cc:1428] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:12.144942 23841 raft_consensus.cc:482] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:12.144987 23841 raft_consensus.cc:504] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.145110 23841 leader_election.cc:288] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267)
I20260927 07:04:12.145516 23837 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Bootstrap starting.
I20260927 07:04:12.145852 23837 tablet_bootstrap.cc:654] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:12.145901 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d34ebd170e47ba8b2715768236c201" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad" is_pre_election: true
I20260927 07:04:12.146239 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d34ebd170e47ba8b2715768236c201" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce" is_pre_election: true
I20260927 07:04:12.146332 23395 raft_consensus.cc:2454] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 0.
I20260927 07:04:12.146536 23837 tablet_bootstrap.cc:492] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: No bootstrap required, opened a new log
W20260927 07:04:12.146526 23606 leader_election.cc:341] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267): Illegal state: must be running to vote when last-logged opid is not known
I20260927 07:04:12.146584 23837 ts_tablet_manager.cc:1397] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:12.146617 23609 leader_election.cc:302] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c2d52f6204547d2bf5a231e76566cce, ea424a0dbe09427b86af0dc7aab75e34; no voters: 88b20e3c35864595bf37106859fc85ad
I20260927 07:04:12.146723 23841 raft_consensus.cc:2781] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:12.146776 23841 raft_consensus.cc:482] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:12.146682 23837 raft_consensus.cc:348] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.146797 23841 raft_consensus.cc:3037] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.146823 23837 raft_consensus.cc:374] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:12.146862 23837 raft_consensus.cc:729] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b20e3c35864595bf37106859fc85ad, State: Initialized, Role: FOLLOWER
I20260927 07:04:12.146911 23837 consensus_queue.cc:260] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.147006 23837 ts_tablet_manager.cc:1428] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:12.147557 23841 raft_consensus.cc:504] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.147676 23841 leader_election.cc:288] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 election: Requested vote from peers 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993), 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267)
I20260927 07:04:12.147778 23826 ts_tablet_manager.cc:1918] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:04:12.147831 23826 log.cc:1196] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/8e01ca665e654928817f82608e0ed55c
I20260927 07:04:12.148090 23826 ts_tablet_manager.cc:1939] T 8e01ca665e654928817f82608e0ed55c P ea424a0dbe09427b86af0dc7aab75e34: Deleting consensus metadata
I20260927 07:04:12.148264 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d34ebd170e47ba8b2715768236c201" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b20e3c35864595bf37106859fc85ad"
I20260927 07:04:12.148329 23535 raft_consensus.cc:3037] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.148406 23222 catalog_manager.cc:4841] TS ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483): tablet 8e01ca665e654928817f82608e0ed55c (table test-table [id=ac9f3975414c4382be802e6d42acea72]) successfully deleted
I20260927 07:04:12.148432 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d34ebd170e47ba8b2715768236c201" candidate_uuid: "ea424a0dbe09427b86af0dc7aab75e34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2d52f6204547d2bf5a231e76566cce"
I20260927 07:04:12.148501 23395 raft_consensus.cc:3037] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:12.148975 23535 raft_consensus.cc:2454] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 1.
I20260927 07:04:12.149206 23395 raft_consensus.cc:2454] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea424a0dbe09427b86af0dc7aab75e34 in term 1.
I20260927 07:04:12.149318 23606 leader_election.cc:302] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b20e3c35864595bf37106859fc85ad, ea424a0dbe09427b86af0dc7aab75e34; no voters:
I20260927 07:04:12.149446 23841 raft_consensus.cc:2781] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:12.149497 23841 raft_consensus.cc:686] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 LEADER]: Becoming Leader. State: Replica: ea424a0dbe09427b86af0dc7aab75e34, State: Running, Role: LEADER
I20260927 07:04:12.149544 23841 consensus_queue.cc:237] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } }
I20260927 07:04:12.149995 23238 catalog_manager.cc:5495] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "ea424a0dbe09427b86af0dc7aab75e34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea424a0dbe09427b86af0dc7aab75e34" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45483 } health_report { overall_health: HEALTHY } } }
I20260927 07:04:12.158370 23535 raft_consensus.cc:1261] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Refusing update from remote peer ea424a0dbe09427b86af0dc7aab75e34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:12.158370 23395 raft_consensus.cc:1261] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce [term 1 FOLLOWER]: Refusing update from remote peer ea424a0dbe09427b86af0dc7aab75e34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:12.158697 23855 consensus_queue.cc:1035] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b20e3c35864595bf37106859fc85ad" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:12.158820 23841 consensus_queue.cc:1035] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2d52f6204547d2bf5a231e76566cce" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 07:04:12.163441 23806 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 07:04:13.681445 23238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:13.681612 23238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:13.682483 23238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72fc7f8aff504da6809b20881ef84142: Sending DeleteTablet for 3 replicas of tablet 83d34ebd170e47ba8b2715768236c201
I20260927 07:04:13.682735 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d34ebd170e47ba8b2715768236c201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:13 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:53054
I20260927 07:04:13.682839 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d34ebd170e47ba8b2715768236c201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:13 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55742
I20260927 07:04:13.682839 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d34ebd170e47ba8b2715768236c201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:13 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:35602
I20260927 07:04:13.683012 23887 tablet_replica.cc:330] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: stopping tablet replica
I20260927 07:04:13.683027 23888 tablet_replica.cc:330] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: stopping tablet replica
I20260927 07:04:13.683013 23886 tablet_replica.cc:330] T 83d34ebd170e47ba8b2715768236c201 P 4c2d52f6204547d2bf5a231e76566cce: stopping tablet replica
I20260927 07:04:13.683072 23888 raft_consensus.cc:2229] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:13.683118 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 23310
I20260927 07:04:13.683109 23887 raft_consensus.cc:2229] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:13.683156 23888 raft_consensus.cc:2258] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:13.683187 23887 raft_consensus.cc:2258] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:13.684135 23888 ts_tablet_manager.cc:1905] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:13.684170 23887 ts_tablet_manager.cc:1905] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:13.686246 23888 ts_tablet_manager.cc:1918] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:04:13.686249 23887 ts_tablet_manager.cc:1918] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:04:13.686301 23887 log.cc:1196] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/83d34ebd170e47ba8b2715768236c201
I20260927 07:04:13.686301 23888 log.cc:1196] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/83d34ebd170e47ba8b2715768236c201
I20260927 07:04:13.686537 23887 ts_tablet_manager.cc:1939] T 83d34ebd170e47ba8b2715768236c201 P 88b20e3c35864595bf37106859fc85ad: Deleting consensus metadata
I20260927 07:04:13.686542 23888 ts_tablet_manager.cc:1939] T 83d34ebd170e47ba8b2715768236c201 P ea424a0dbe09427b86af0dc7aab75e34: Deleting consensus metadata
I20260927 07:04:13.687031 23222 catalog_manager.cc:4841] TS 88b20e3c35864595bf37106859fc85ad (127.21.2.66:37267): tablet 83d34ebd170e47ba8b2715768236c201 (table test-table [id=b84b60deaa8f4cf7a06bd5e98d33c9b4]) successfully deleted
I20260927 07:04:13.687143 23222 catalog_manager.cc:4841] TS ea424a0dbe09427b86af0dc7aab75e34 (127.21.2.67:45483): tablet 83d34ebd170e47ba8b2715768236c201 (table test-table [id=b84b60deaa8f4cf7a06bd5e98d33c9b4]) successfully deleted
W20260927 07:04:13.691043 23225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:41993 (error 108)
I20260927 07:04:13.691269 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 23450
I20260927 07:04:13.691972 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:13.691505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41993 (local address 127.0.0.1:53082)
0927 07:04:13.691879 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:13.691925 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":64,"threads_started":1}
W20260927 07:04:13.692128 23225 catalog_manager.cc:4584] TS 4c2d52f6204547d2bf5a231e76566cce (127.21.2.65:41993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83d34ebd170e47ba8b2715768236c201: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41993: connect: Connection refused (error 111)
I20260927 07:04:13.698556 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 23590
I20260927 07:04:13.704246 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 23206
2026-09-27T07:04:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5800 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:13 dist-test-slave-ltqz krb5kdc[23893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:13 dist-test-slave-ltqz krb5kdc[23893](info): set up 2 sockets
Sep 27 07:04:13 dist-test-slave-ltqz krb5kdc[23893](info): commencing operation
krb5kdc: starting...
W20260927 07:04:14.751077 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:14 dist-test-slave-ltqz krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:14.870937 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46541
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:14.952114 23906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:14.952313 23906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:14.952337 23906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:14.953464 23906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:14.953504 23906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:14.953518 23906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:14.953531 23906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:14.953543 23906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:14.954977 23906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:14.955214 23906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:14.955482 23906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:14.960161 23911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:14.960170 23914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:14.960173 23912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:14.960474 23906 server_base.cc:1034] running on GCE node
I20260927 07:04:14.960705 23906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:14.961028 23906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:14.962208 23906 hybrid_clock.cc:648] HybridClock initialized: now 1790492654962191 us; error 34 us; skew 500 ppm
Sep 27 07:04:14 dist-test-slave-ltqz krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:14.964654 23906 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:14.965083 23906 webserver.cc:460] Webserver started at http://127.21.2.126:41433/ using document root <none> and password file <none>
I20260927 07:04:14.965322 23906 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:14.965392 23906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:14.965543 23906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:14.966578 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "f36f7b83498c4464b052671b115de1ba"
format_stamp: "Formatted at 2026-09-27 07:04:14 on dist-test-slave-ltqz"
server_key: "e26d6f293006315ed97d7ac524479290"
server_key_iv: "3aacffa2ad7b995b6607aeaefa37af6c"
server_key_version: "encryptionkey@0"
I20260927 07:04:14.966943 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "f36f7b83498c4464b052671b115de1ba"
format_stamp: "Formatted at 2026-09-27 07:04:14 on dist-test-slave-ltqz"
server_key: "e26d6f293006315ed97d7ac524479290"
server_key_iv: "3aacffa2ad7b995b6607aeaefa37af6c"
server_key_version: "encryptionkey@0"
I20260927 07:04:14.968304 23906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:14.968988 23921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:14.969166 23906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:14.969262 23906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "f36f7b83498c4464b052671b115de1ba"
format_stamp: "Formatted at 2026-09-27 07:04:14 on dist-test-slave-ltqz"
server_key: "e26d6f293006315ed97d7ac524479290"
server_key_iv: "3aacffa2ad7b995b6607aeaefa37af6c"
server_key_version: "encryptionkey@0"
I20260927 07:04:14.969347 23906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:14.981976 23906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:14.984040 23906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:14.984217 23906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:14.988216 23906 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:46541
I20260927 07:04:14.988270 23973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:46541 every 8 connection(s)
I20260927 07:04:14.988638 23906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:14.989678 23974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:14.993129 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: Bootstrap starting.
I20260927 07:04:14.993850 23974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:14.994185 23974 log.cc:824] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:14.995013 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: No bootstrap required, opened a new log
I20260927 07:04:14.995874 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 23906
I20260927 07:04:14.995972 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:14.996150 21513 external_mini_cluster.cc:1428] Setting key c84745031a2c1b74f35750ef0e6db8ba
I20260927 07:04:14.997365 23974 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } }
I20260927 07:04:14.997526 23974 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:14.997591 23974 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f7b83498c4464b052671b115de1ba, State: Initialized, Role: FOLLOWER
I20260927 07:04:14.997721 23974 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } }
I20260927 07:04:14.997824 23974 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:14.997876 23974 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:14.997926 23974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:14.999289 23974 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } }
I20260927 07:04:14.999442 23974 leader_election.cc:302] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36f7b83498c4464b052671b115de1ba; no voters:
I20260927 07:04:14.999641 23974 leader_election.cc:288] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:14.999693 23979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:14.999864 23979 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [term 1 LEADER]: Becoming Leader. State: Replica: f36f7b83498c4464b052671b115de1ba, State: Running, Role: LEADER
I20260927 07:04:14.999997 23979 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } }
I20260927 07:04:15.000005 23974 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:15.000451 23981 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36f7b83498c4464b052671b115de1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } } }
I20260927 07:04:15.000466 23980 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36f7b83498c4464b052671b115de1ba. Latest consensus state: current_term: 1 leader_uuid: "f36f7b83498c4464b052671b115de1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f7b83498c4464b052671b115de1ba" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46541 } } }
I20260927 07:04:15.000579 23981 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:15.000584 23980 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:15.002198 23995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:15.002259 23995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:15.002300 23985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:15.002502 23985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:15.005021 23985 catalog_manager.cc:1347] Generated new cluster ID: 4b085e5650bd4c5890ed82a6f54b7ff2
I20260927 07:04:15.005077 23985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:15.009109 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:14.996663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.21.2.126:46541)
0927 07:04:14.996865 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:14.996875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:14.996910 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:04:14.997342 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:14.997348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:14.997393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:14.997537 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:14.998278 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:14.999427 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.000546 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.001167 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.001275 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.005212 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:15.005216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:15.005217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:15.005230 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:15.007334 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.007866 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.007870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.007870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.008002 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.008244 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.008245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.008245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.008505 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:15.008671 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:15.008770 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 07:04:15.008918 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260927 07:04:15.031709 23985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:15.032373 23985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:15.036109 23985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36f7b83498c4464b052671b115de1ba: Generated new TSK 0
I20260927 07:04:15.036262 23985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:15.068133 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:15.152499 24000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:15.152751 24000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:15.152791 24000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:15.153985 24000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:15.154070 24000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:15.154119 24000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:15.155645 24000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:15.155915 24000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:15.156152 24000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:15.161160 24005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:15.161196 24008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:15.161237 24006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:15.161310 24000 server_base.cc:1034] running on GCE node
I20260927 07:04:15.161589 24000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:15.161841 24000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:15.163028 24000 hybrid_clock.cc:648] HybridClock initialized: now 1790492655162996 us; error 37 us; skew 500 ppm
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:15.165683 24000 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:15.166184 24000 webserver.cc:460] Webserver started at http://127.21.2.65:42423/ using document root <none> and password file <none>
I20260927 07:04:15.166365 24000 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:15.166404 24000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:15.166500 24000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:15.167580 24000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "848bc114f6754bd5bb059dc8ccdd428b"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "bd2dde155a9b4a93310e2e109168a05e"
server_key_iv: "95c0a6d9819316c694f6294864bb22c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.167886 24000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "848bc114f6754bd5bb059dc8ccdd428b"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "bd2dde155a9b4a93310e2e109168a05e"
server_key_iv: "95c0a6d9819316c694f6294864bb22c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.169169 24000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:15.169873 24015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.170074 24000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:15.170147 24000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "848bc114f6754bd5bb059dc8ccdd428b"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "bd2dde155a9b4a93310e2e109168a05e"
server_key_iv: "95c0a6d9819316c694f6294864bb22c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.170238 24000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:15.188506 24000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:15.190311 24000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:15.190477 24000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:15.190726 24000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:15.191044 24000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:15.191078 24000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.191100 24000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:15.191150 24000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.196763 24000 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37565
I20260927 07:04:15.196805 24128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37565 every 8 connection(s)
I20260927 07:04:15.197160 24000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:15.203567 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24000
I20260927 07:04:15.203641 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:15.203841 21513 external_mini_cluster.cc:1428] Setting key 9707f43f70b160b91b24043abb428a74
I20260927 07:04:15.207257 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.197893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:60163 (local address 127.21.2.126:46541)
0927 07:04:15.197960 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:15.197962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:15.198161 (+ 199us) server_negotiation.cc:407] Connection header received
0927 07:04:15.198629 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:15.198630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:15.198643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:15.198702 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:15.199693 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.200260 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.201447 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.201661 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.201744 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.203989 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:15.203991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:15.203991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:15.204000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:15.205810 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.206229 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.206230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.206231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.206300 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.206540 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.206542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.206542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.206691 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:15.206763 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:15.207019 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:04:15.207117 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:04:15.207267 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.197850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46541 (local address 127.21.2.65:60163)
0927 07:04:15.198031 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:15.198073 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:15.198546 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:15.198770 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:15.198773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:15.199338 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:15.199619 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.199624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.200356 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.200357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.201371 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.201374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.201767 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.201769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.201900 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.202383 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:15.202390 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:15.203848 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:15.205914 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.205920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.205922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.206160 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.206373 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.206374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.206374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.206473 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.206757 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:15.206761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:15.206873 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 07:04:15.206977 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:04:15.207097 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260927 07:04:15.207880 24129 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46541
I20260927 07:04:15.208001 24129 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:15.208171 24129 heartbeater.cc:511] Master 127.21.2.126:46541 requested a full tablet report, sending...
I20260927 07:04:15.208698 23938 ts_manager.cc:194] Registered new tserver with Master: 848bc114f6754bd5bb059dc8ccdd428b (127.21.2.65:37565)
I20260927 07:04:15.209689 23938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:60163
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:15.230222 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:15.310938 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:15.311146 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:15.311170 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:15.312317 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:15.312357 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:15.312384 24134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:15.313748 24134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:15.313934 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:15.314121 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:15.318846 24139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:15.318899 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:15.318852 24140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:15.319072 24134 server_base.cc:1034] running on GCE node
I20260927 07:04:15.319438 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:15.319698 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:15.320865 24134 hybrid_clock.cc:648] HybridClock initialized: now 1790492655320853 us; error 32 us; skew 500 ppm
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:15.323503 24134 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:15.324002 24134 webserver.cc:460] Webserver started at http://127.21.2.66:39535/ using document root <none> and password file <none>
I20260927 07:04:15.324259 24134 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:15.324327 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:15.324463 24134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:15.325511 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "ec39ff18148b4b38ac89a5a3d96d88af"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "ad2e54aed0d7a0a69be87ff54499304b"
server_key_iv: "f29b0f0d479c6acb175f9dd602a22d82"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.325881 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "ec39ff18148b4b38ac89a5a3d96d88af"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "ad2e54aed0d7a0a69be87ff54499304b"
server_key_iv: "f29b0f0d479c6acb175f9dd602a22d82"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.327212 24134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:15.328130 24149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.328366 24134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:15.328475 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "ec39ff18148b4b38ac89a5a3d96d88af"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "ad2e54aed0d7a0a69be87ff54499304b"
server_key_iv: "f29b0f0d479c6acb175f9dd602a22d82"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.328567 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:15.343881 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:15.346010 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:15.346185 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:15.346433 24134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:15.346736 24134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:15.346791 24134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.346835 24134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:15.346858 24134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.352938 24134 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46691
I20260927 07:04:15.353018 24262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46691 every 8 connection(s)
I20260927 07:04:15.353415 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:15.355301 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24134
I20260927 07:04:15.355417 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:15.355613 21513 external_mini_cluster.cc:1428] Setting key 87047e84fafd8a8cb1c255df6eb31a61
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
I20260927 07:04:15.363960 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.354590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:55001 (local address 127.21.2.126:46541)
0927 07:04:15.354640 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:15.354642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:15.354652 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:15.355253 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:15.355255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:15.355268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:15.355302 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:15.356603 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.357214 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.358342 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.358558 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.358651 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.361125 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:15.361126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:15.361127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:15.361134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:15.362532 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.362940 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.362941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.362941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.363023 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.363258 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.363259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.363260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.363427 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:15.363513 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:15.363778 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 07:04:15.363869 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:04:15.364180 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.354272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46541 (local address 127.21.2.66:55001)
0927 07:04:15.354441 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:15.354478 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:15.355120 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:15.355545 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:15.355549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:15.356254 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:15.356512 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.356517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.357312 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.357313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.358249 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.358251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.358659 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.358660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.358785 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.359315 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:15.359323 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:15.361015 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:15.362634 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.362640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.362642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.362860 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.363077 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.363078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.363079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.363187 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.363503 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:15.363509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:15.363647 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:04:15.363758 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:04:15.363914 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260927 07:04:15.364696 24263 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46541
I20260927 07:04:15.364794 24263 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:15.365001 24263 heartbeater.cc:511] Master 127.21.2.126:46541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
I20260927 07:04:15.365468 23938 ts_manager.cc:194] Registered new tserver with Master: ec39ff18148b4b38ac89a5a3d96d88af (127.21.2.66:46691)
I20260927 07:04:15.365922 23938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:55001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:15.380298 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:15.460305 24268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:15.460510 24268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:15.460532 24268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:15.461617 24268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:15.461651 24268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:15.461678 24268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:15.462983 24268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:15.463153 24268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:15.463315 24268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:15.468146 24274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:15.468196 24273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:15.468338 24268 server_base.cc:1034] running on GCE node
W20260927 07:04:15.468214 24276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:15.468587 24268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:15.468832 24268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:15.470000 24268 hybrid_clock.cc:648] HybridClock initialized: now 1790492655469992 us; error 31 us; skew 500 ppm
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:15.472517 24268 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:15.472985 24268 webserver.cc:460] Webserver started at http://127.21.2.67:39733/ using document root <none> and password file <none>
I20260927 07:04:15.473212 24268 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:15.473311 24268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:15.473469 24268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:15.474609 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "e57853dd353543d785054d0e40f8ae59"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "c525ddefc113b55e3f7d6298ce279f78"
server_key_iv: "f4681a2f95bd0346b1fe41b6b39118db"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.475000 24268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "e57853dd353543d785054d0e40f8ae59"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "c525ddefc113b55e3f7d6298ce279f78"
server_key_iv: "f4681a2f95bd0346b1fe41b6b39118db"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.476438 24268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:15.477269 24283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.477566 24268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:15.477638 24268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "e57853dd353543d785054d0e40f8ae59"
format_stamp: "Formatted at 2026-09-27 07:04:15 on dist-test-slave-ltqz"
server_key: "c525ddefc113b55e3f7d6298ce279f78"
server_key_iv: "f4681a2f95bd0346b1fe41b6b39118db"
server_key_version: "encryptionkey@0"
I20260927 07:04:15.477728 24268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:15.492854 24268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:15.494858 24268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:15.495018 24268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:15.495271 24268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:15.495644 24268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:15.495702 24268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.495745 24268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:15.495774 24268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:15.501850 24268 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:42993
I20260927 07:04:15.501945 24396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:42993 every 8 connection(s)
I20260927 07:04:15.502255 24268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:15.505055 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24268
I20260927 07:04:15.505175 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:15.505348 21513 external_mini_cluster.cc:1428] Setting key ef0ff7c5eb399f74155748b2e40db552
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[23893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:15.512094 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.502988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48993 (local address 127.21.2.126:46541)
0927 07:04:15.503030 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:15.503033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:15.503211 (+ 178us) server_negotiation.cc:407] Connection header received
0927 07:04:15.503806 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:15.503807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:15.503819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:15.503903 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:15.504838 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.505455 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.506647 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.506855 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.506947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.509204 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:15.509212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:15.509213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:15.509220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:15.510752 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.511142 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.511142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.511143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.511193 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:15.511561 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:15.511561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:15.511562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:15.511666 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:15.511719 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:15.511966 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 07:04:15.512030 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:04:15.512286 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:15.502944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46541 (local address 127.21.2.67:48993)
0927 07:04:15.503081 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:15.503129 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:15.503701 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:15.503969 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:15.503973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:15.504543 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:15.504729 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.504733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.505550 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.505551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.506561 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:15.506563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:15.506956 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:15.506957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:15.507091 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:15.507601 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:15.507609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:15.509079 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:15.510841 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.510847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.510849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.511074 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.511253 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:15.511253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:15.511253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:15.511345 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:15.511726 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:15.511731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:15.511845 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 07:04:15.511921 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:04:15.512049 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260927 07:04:15.512826 24397 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46541
I20260927 07:04:15.512928 24397 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:15.513104 24397 heartbeater.cc:511] Master 127.21.2.126:46541 requested a full tablet report, sending...
I20260927 07:04:15.513460 23938 ts_manager.cc:194] Registered new tserver with Master: e57853dd353543d785054d0e40f8ae59 (127.21.2.67:42993)
I20260927 07:04:15.513828 23938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:48993
I20260927 07:04:15.517197 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:04:15.528532 23978 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:15.527499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.21.2.126:46541)
0927 07:04:15.527565 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:15.527567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:15.527716 (+ 149us) server_negotiation.cc:407] Connection header received
0927 07:04:15.527810 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:15.527811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:15.527827 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:15.527881 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:15.528453 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45466: BlockingRecv error: recv got EOF from 127.0.0.1:45466 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:04:15.529340 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24000
I20260927 07:04:15.533860 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24134
I20260927 07:04:15.537962 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24268
I20260927 07:04:15.542232 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 23906
2026-09-27T07:04:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1837 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[24409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[24409](info): set up 2 sockets
Sep 27 07:04:15 dist-test-slave-ltqz krb5kdc[24409](info): commencing operation
krb5kdc: starting...
W20260927 07:04:16.580051 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:16.702353 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:37269
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:16.783712 24422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:16.783960 24422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:16.783985 24422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:16.785152 24422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:16.785197 24422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:16.785211 24422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:16.785224 24422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:16.785238 24422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:16.786855 24422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:37269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:16.787124 24422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:16.787351 24422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:16.792172 24427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:16.792186 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:16.792188 24428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:16.792562 24422 server_base.cc:1034] running on GCE node
I20260927 07:04:16.792822 24422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:16.793120 24422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:16.794315 24422 hybrid_clock.cc:648] HybridClock initialized: now 1790492656794298 us; error 34 us; skew 500 ppm
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492656, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:16.796967 24422 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:16.797402 24422 webserver.cc:460] Webserver started at http://127.21.2.126:38627/ using document root <none> and password file <none>
I20260927 07:04:16.797645 24422 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:16.797690 24422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:16.797853 24422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:16.798907 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "939039edaf244827b1075fa6d49d022b"
format_stamp: "Formatted at 2026-09-27 07:04:16 on dist-test-slave-ltqz"
server_key: "14725d36665a799beaa2a458da52975b"
server_key_iv: "09de1521948d174d271f4ef69fdc3c74"
server_key_version: "encryptionkey@0"
I20260927 07:04:16.799274 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "939039edaf244827b1075fa6d49d022b"
format_stamp: "Formatted at 2026-09-27 07:04:16 on dist-test-slave-ltqz"
server_key: "14725d36665a799beaa2a458da52975b"
server_key_iv: "09de1521948d174d271f4ef69fdc3c74"
server_key_version: "encryptionkey@0"
I20260927 07:04:16.800637 24422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:16.801424 24437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:16.801637 24422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:16.801723 24422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "939039edaf244827b1075fa6d49d022b"
format_stamp: "Formatted at 2026-09-27 07:04:16 on dist-test-slave-ltqz"
server_key: "14725d36665a799beaa2a458da52975b"
server_key_iv: "09de1521948d174d271f4ef69fdc3c74"
server_key_version: "encryptionkey@0"
I20260927 07:04:16.801776 24422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:16.823722 24422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:16.825631 24422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:16.825742 24422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:16.829456 24422 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:37269
I20260927 07:04:16.829537 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:37269 every 8 connection(s)
I20260927 07:04:16.830015 24422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:16.830678 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:16.833016 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: Bootstrap starting.
I20260927 07:04:16.833593 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:16.833879 24490 log.cc:824] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:16.834599 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: No bootstrap required, opened a new log
I20260927 07:04:16.836279 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } }
I20260927 07:04:16.836441 24490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:16.836490 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939039edaf244827b1075fa6d49d022b, State: Initialized, Role: FOLLOWER
I20260927 07:04:16.836669 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } }
I20260927 07:04:16.836768 24490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:16.836812 24490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:16.836861 24490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:16.837249 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24422
I20260927 07:04:16.837352 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:16.837561 21513 external_mini_cluster.cc:1428] Setting key 3e58771c4c7053b1c0888e72f078bd71
I20260927 07:04:16.838102 24490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } }
I20260927 07:04:16.838244 24490 leader_election.cc:302] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 939039edaf244827b1075fa6d49d022b; no voters:
I20260927 07:04:16.838421 24490 leader_election.cc:288] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:16.838454 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:16.838606 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [term 1 LEADER]: Becoming Leader. State: Replica: 939039edaf244827b1075fa6d49d022b, State: Running, Role: LEADER
I20260927 07:04:16.838689 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:16.838752 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } }
I20260927 07:04:16.839071 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "939039edaf244827b1075fa6d49d022b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } } }
I20260927 07:04:16.839239 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:16.839444 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939039edaf244827b1075fa6d49d022b. Latest consensus state: current_term: 1 leader_uuid: "939039edaf244827b1075fa6d49d022b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939039edaf244827b1075fa6d49d022b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37269 } } }
I20260927 07:04:16.839568 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:16.840896 24511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:16.840952 24511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:16.841001 24504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:16.841181 24504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:16.842646 24504 catalog_manager.cc:1347] Generated new cluster ID: 91247917c563465bb1e04ff78ffdb480
I20260927 07:04:16.842700 24504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:16.848904 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:16.838032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.21.2.126:37269)
0927 07:04:16.838229 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:16.838240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:16.838274 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:04:16.838396 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:16.838401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:16.838444 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:16.838594 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:16.839486 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:16.841266 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:16.842033 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:16.842596 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:16.842704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:16.845241 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:16.845245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:16.845245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:16.845257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:16.847195 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:16.847648 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:16.847651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:16.847652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:16.847772 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:16.847994 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:16.847995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:16.847995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:16.848288 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:16.848432 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:16.848556 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 07:04:16.848689 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260927 07:04:16.850175 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:04:16.864969 24504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:16.865728 24504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:16.869740 24504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 939039edaf244827b1075fa6d49d022b: Generated new TSK 0
I20260927 07:04:16.869936 24504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:04:16.871095 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:16.867487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.21.2.126:37269)
0927 07:04:16.867548 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:16.867551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:16.867725 (+ 174us) server_negotiation.cc:407] Connection header received
0927 07:04:16.867797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:16.867798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:16.867810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:16.867864 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:16.868236 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:16.868731 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:16.869566 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:16.869866 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:16.869927 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:16.870682 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:16.870684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:16.870684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:16.870857 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:16.870873 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:16.870879 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:16.870928 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:16.871018 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:04:16.871042 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:04:16.873579 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24422
2026-09-27T07:04:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1327 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24523](info): set up 2 sockets
Sep 27 07:04:16 dist-test-slave-ltqz krb5kdc[24523](info): commencing operation
krb5kdc: starting...
W20260927 07:04:17.911243 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:17 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:18.028087 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40603
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:18.107568 24536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:18.107767 24536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:18.107788 24536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:18.108952 24536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:18.108995 24536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:18.109009 24536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:18.109022 24536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:18.109081 24536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:18.110523 24536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:18.110795 24536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:18.111019 24536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:18.115816 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.115787 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.115797 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:18.115967 24536 server_base.cc:1034] running on GCE node
I20260927 07:04:18.116276 24536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:18.116587 24536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:18.117758 24536 hybrid_clock.cc:648] HybridClock initialized: now 1790492658117750 us; error 51 us; skew 500 ppm
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:18.120402 24536 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:18.120915 24536 webserver.cc:460] Webserver started at http://127.21.2.126:38217/ using document root <none> and password file <none>
I20260927 07:04:18.121168 24536 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:18.121239 24536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:18.121388 24536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:18.122469 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "7b6834ba59c22e891054f125b96af5d4"
server_key_iv: "a1f98791bd2e1160f279932dcfcb91c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.122843 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "7b6834ba59c22e891054f125b96af5d4"
server_key_iv: "a1f98791bd2e1160f279932dcfcb91c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.124257 24536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:18.125036 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.125243 24536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:18.125339 24536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "7b6834ba59c22e891054f125b96af5d4"
server_key_iv: "a1f98791bd2e1160f279932dcfcb91c5"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.125411 24536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:18.137127 24536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:18.138967 24536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:18.139129 24536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:18.142881 24536 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40603
I20260927 07:04:18.142949 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40603 every 8 connection(s)
I20260927 07:04:18.143280 24536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:18.144304 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:18.146486 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232: Bootstrap starting.
I20260927 07:04:18.147063 24604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:18.147343 24604 log.cc:824] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:18.148052 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232: No bootstrap required, opened a new log
I20260927 07:04:18.149550 24604 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } }
I20260927 07:04:18.149683 24604 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:18.149706 24604 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc23baa2e7b4b0088ae58b4d0f7a232, State: Initialized, Role: FOLLOWER
I20260927 07:04:18.149840 24604 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } }
I20260927 07:04:18.149922 24604 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:18.149945 24604 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:18.149981 24604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:18.151079 24604 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } }
I20260927 07:04:18.151175 24604 leader_election.cc:302] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecc23baa2e7b4b0088ae58b4d0f7a232; no voters:
I20260927 07:04:18.151301 24604 leader_election.cc:288] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:18.151350 24607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:18.151554 24607 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [term 1 LEADER]: Becoming Leader. State: Replica: ecc23baa2e7b4b0088ae58b4d0f7a232, State: Running, Role: LEADER
I20260927 07:04:18.151628 24604 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:18.151687 24607 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } }
I20260927 07:04:18.152076 24609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecc23baa2e7b4b0088ae58b4d0f7a232. Latest consensus state: current_term: 1 leader_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } } }
I20260927 07:04:18.152166 24609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:18.152064 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc23baa2e7b4b0088ae58b4d0f7a232" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40603 } } }
I20260927 07:04:18.152429 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:18.152547 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24536
I20260927 07:04:18.152654 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:18.152671 24615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:18.152839 21513 external_mini_cluster.cc:1428] Setting key 51421e9073e804a33a7edb0f9340dffe
I20260927 07:04:18.153283 24615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:18.154857 24615 catalog_manager.cc:1347] Generated new cluster ID: 9941887d4da94f93b1ab3a4cfffd4309
I20260927 07:04:18.154913 24615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:18.163688 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.153342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.21.2.126:40603)
0927 07:04:18.153491 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:18.153501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:18.154115 (+ 614us) server_negotiation.cc:407] Connection header received
0927 07:04:18.154236 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:18.154244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:18.154320 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:18.154467 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:18.155196 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.156204 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.156985 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.157430 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.157549 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.159931 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:18.159936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:18.159936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:18.159948 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:18.162015 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.162409 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.162412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.162413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.162519 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.162746 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.162747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.162747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.162987 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:18.163171 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:18.163235 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 07:04:18.163498 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260927 07:04:18.164503 24615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:18.165210 24615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:18.173285 24615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecc23baa2e7b4b0088ae58b4d0f7a232: Generated new TSK 0
I20260927 07:04:18.173480 24615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:18.210846 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:18.327051 24630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:18.327319 24630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:18.327354 24630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:18.329150 24630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:18.329216 24630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:18.329265 24630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:18.331281 24630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:18.331557 24630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:18.331800 24630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:18.336846 24638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.336917 24636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.336859 24635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:18.337157 24630 server_base.cc:1034] running on GCE node
I20260927 07:04:18.337365 24630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:18.337631 24630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:18.338781 24630 hybrid_clock.cc:648] HybridClock initialized: now 1790492658338767 us; error 29 us; skew 500 ppm
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:18.341354 24630 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:18.341830 24630 webserver.cc:460] Webserver started at http://127.21.2.65:36465/ using document root <none> and password file <none>
I20260927 07:04:18.342079 24630 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:18.342149 24630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:18.342296 24630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:18.343345 24630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "cb4a2d99befb41c19feaea3beb9dca57"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "6989fa63e091ed36cd1ddb57e6ae2d8f"
server_key_iv: "6be0f654eb0e8b413f48173f1277a8db"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.343797 24630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "cb4a2d99befb41c19feaea3beb9dca57"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "6989fa63e091ed36cd1ddb57e6ae2d8f"
server_key_iv: "6be0f654eb0e8b413f48173f1277a8db"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.345266 24630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:04:18.346010 24645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.346206 24630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:18.346314 24630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "cb4a2d99befb41c19feaea3beb9dca57"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "6989fa63e091ed36cd1ddb57e6ae2d8f"
server_key_iv: "6be0f654eb0e8b413f48173f1277a8db"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.346390 24630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:18.356470 24630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:18.358557 24630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:18.358707 24630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:18.358945 24630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:18.359258 24630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:18.359293 24630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.359335 24630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:18.359385 24630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.365698 24630 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:40197
I20260927 07:04:18.365820 24758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:40197 every 8 connection(s)
I20260927 07:04:18.366128 24630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:18.366705 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24630
I20260927 07:04:18.366796 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:18.367014 21513 external_mini_cluster.cc:1428] Setting key 43a3d049cabbc71ce737f17dcc8407a5
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:04:18.376513 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.367197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:41717 (local address 127.21.2.126:40603)
0927 07:04:18.367303 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:18.367306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:18.367319 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:04:18.368022 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:18.368023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:18.368041 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:18.368101 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:18.369043 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.369697 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.370980 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.371384 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.371458 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.373556 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:18.373558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:18.373558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:18.373566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:18.375204 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.375678 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.375680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.375680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.375742 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.375984 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.375985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.375985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.376098 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:18.376154 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:18.376382 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 07:04:18.376448 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:04:18.376713 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.366896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40603 (local address 127.21.2.65:41717)
0927 07:04:18.367164 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:18.367231 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:18.367911 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:18.368180 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:18.368184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:18.368754 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:18.368941 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.368945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.369791 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.369792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.370888 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.370891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.371482 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.371483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.371596 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.372143 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:18.372152 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:18.373464 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:18.375284 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.375290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.375291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.375576 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.375816 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.375817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.375817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.375914 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.376159 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:18.376163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:18.376272 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 07:04:18.376352 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:04:18.376465 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260927 07:04:18.377178 24759 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40603
I20260927 07:04:18.377280 24759 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:18.377450 24759 heartbeater.cc:511] Master 127.21.2.126:40603 requested a full tablet report, sending...
I20260927 07:04:18.378000 24568 ts_manager.cc:194] Registered new tserver with Master: cb4a2d99befb41c19feaea3beb9dca57 (127.21.2.65:40197)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
I20260927 07:04:18.378667 24568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:41717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:18.394450 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:18.474893 24764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:18.475158 24764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:18.475198 24764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:18.476430 24764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:18.476508 24764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:18.476555 24764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:18.477998 24764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:18.478263 24764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:18.478495 24764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:18.483527 24769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.483559 24772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.483527 24770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:18.483809 24764 server_base.cc:1034] running on GCE node
I20260927 07:04:18.484014 24764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:18.484268 24764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:18.485463 24764 hybrid_clock.cc:648] HybridClock initialized: now 1790492658485447 us; error 30 us; skew 500 ppm
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:18.487975 24764 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:18.488395 24764 webserver.cc:460] Webserver started at http://127.21.2.66:33823/ using document root <none> and password file <none>
I20260927 07:04:18.488627 24764 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:18.488694 24764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:18.488858 24764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:18.489915 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "7a978d158652420a8ebafeb8d8c1adb2"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "e17d4681907a00fea3b96fcefcd14011"
server_key_iv: "f06ef7b458571089917d882eecc7a7c8"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.490294 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "7a978d158652420a8ebafeb8d8c1adb2"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "e17d4681907a00fea3b96fcefcd14011"
server_key_iv: "f06ef7b458571089917d882eecc7a7c8"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.491709 24764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:18.492435 24779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.492610 24764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:18.492708 24764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "7a978d158652420a8ebafeb8d8c1adb2"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "e17d4681907a00fea3b96fcefcd14011"
server_key_iv: "f06ef7b458571089917d882eecc7a7c8"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.492787 24764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:18.508672 24764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:18.510725 24764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:18.510898 24764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:18.511148 24764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:18.511510 24764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:18.511572 24764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.511616 24764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:18.511647 24764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.517489 24764 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:34607
I20260927 07:04:18.517534 24892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:34607 every 8 connection(s)
I20260927 07:04:18.517860 24764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:18.519325 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24764
I20260927 07:04:18.519464 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:18.519676 21513 external_mini_cluster.cc:1428] Setting key cb576cabba502ad4899345e4d6fb6a3b
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
I20260927 07:04:18.528602 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.518676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:51253 (local address 127.21.2.126:40603)
0927 07:04:18.518725 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:18.518729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:18.518942 (+ 213us) server_negotiation.cc:407] Connection header received
0927 07:04:18.519668 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:18.519669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:18.519680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:18.519767 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:18.520936 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.521423 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.522533 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.522738 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.522829 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.525544 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:18.525545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:18.525546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:18.525552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:18.526901 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.527433 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.527434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.527434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.527499 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.527807 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.527808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.527808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.527917 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:18.527966 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:18.528382 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 07:04:18.528503 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:04:18.528720 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.518636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40603 (local address 127.21.2.66:51253)
0927 07:04:18.518801 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:18.518844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:18.519555 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:18.519854 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:18.519859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:18.520551 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:18.520838 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.520846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.521526 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.521527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.522435 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.522438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.522838 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.522839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.522966 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.523567 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:18.523575 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:18.525428 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:18.526991 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.526997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.527000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.527310 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.527575 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.527576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.527577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.527726 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.527982 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:18.527988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:18.528154 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 07:04:18.528317 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 07:04:18.528500 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260927 07:04:18.529193 24893 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40603
I20260927 07:04:18.529287 24893 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:18.529479 24893 heartbeater.cc:511] Master 127.21.2.126:40603 requested a full tablet report, sending...
I20260927 07:04:18.529860 24568 ts_manager.cc:194] Registered new tserver with Master: 7a978d158652420a8ebafeb8d8c1adb2 (127.21.2.66:34607)
I20260927 07:04:18.530242 24568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:51253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:18.545542 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:18.625857 24898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:18.626091 24898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:18.626127 24898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:18.627281 24898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:18.627349 24898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:18.627414 24898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:18.628907 24898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:18.629168 24898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:18.629451 24898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:18.634267 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.634261 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:18.634311 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:18.634509 24898 server_base.cc:1034] running on GCE node
I20260927 07:04:18.634708 24898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:18.634893 24898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:18.636029 24898 hybrid_clock.cc:648] HybridClock initialized: now 1790492658636016 us; error 28 us; skew 500 ppm
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:18.638451 24898 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:18.638877 24898 webserver.cc:460] Webserver started at http://127.21.2.67:46131/ using document root <none> and password file <none>
I20260927 07:04:18.639111 24898 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:18.639155 24898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:18.639295 24898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:18.640409 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "fff80f400d3a4f25b7a58cb1b62fc171"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "c0e15b71bdecea324f9c3212ce49023f"
server_key_iv: "4ce380c3cb4417e6072089178dac5041"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.640758 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "fff80f400d3a4f25b7a58cb1b62fc171"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "c0e15b71bdecea324f9c3212ce49023f"
server_key_iv: "4ce380c3cb4417e6072089178dac5041"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.642120 24898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:18.642858 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.643070 24898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:18.643148 24898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "fff80f400d3a4f25b7a58cb1b62fc171"
format_stamp: "Formatted at 2026-09-27 07:04:18 on dist-test-slave-ltqz"
server_key: "c0e15b71bdecea324f9c3212ce49023f"
server_key_iv: "4ce380c3cb4417e6072089178dac5041"
server_key_version: "encryptionkey@0"
I20260927 07:04:18.643198 24898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:18.653353 24898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:18.655970 24898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:18.656116 24898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:18.656301 24898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:18.656642 24898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:18.656677 24898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.656700 24898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:18.656752 24898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:18.662474 24898 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:40777
I20260927 07:04:18.662544 25026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:40777 every 8 connection(s)
I20260927 07:04:18.662876 24898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:18.670326 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 24898
I20260927 07:04:18.670442 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:18.670631 21513 external_mini_cluster.cc:1428] Setting key eacb715b97c6c01865b61838e4632815
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:18.675827 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.663679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:46651 (local address 127.21.2.126:40603)
0927 07:04:18.663755 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:18.663758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:18.663966 (+ 208us) server_negotiation.cc:407] Connection header received
0927 07:04:18.664428 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:18.664429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:18.664440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:18.664490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:18.665373 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.665862 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.666944 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.667129 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.667184 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.672625 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:18.672626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:18.672627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:18.672636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:18.674333 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.674774 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.674775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.674776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.674836 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.675106 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.675107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.675107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.675265 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:18.675322 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:18.675651 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 07:04:18.675731 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:04:18.676079 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.663646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40603 (local address 127.21.2.67:46651)
0927 07:04:18.663846 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:18.663882 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:18.664349 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:18.664541 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:18.664544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:18.665101 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:18.665289 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.665293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.665942 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.665942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.666869 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:18.666872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.667217 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.667218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:18.667397 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.668015 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:18.668024 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:18.672522 (+ 4498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:18.674435 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.674442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.674444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.674695 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.674912 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:18.674913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:18.674913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:18.675030 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:18.675325 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:18.675332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:18.675511 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 07:04:18.675630 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 07:04:18.675793 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260927 07:04:18.676577 25027 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40603
I20260927 07:04:18.676685 25027 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:18.676867 25027 heartbeater.cc:511] Master 127.21.2.126:40603 requested a full tablet report, sending...
I20260927 07:04:18.677407 24568 ts_manager.cc:194] Registered new tserver with Master: fff80f400d3a4f25b7a58cb1b62fc171 (127.21.2.67:40777)
I20260927 07:04:18.677788 24568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:46651
I20260927 07:04:18.682593 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[24523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492658, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:18.705559 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:18.698029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.21.2.126:40603)
0927 07:04:18.698159 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:18.698161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:18.698404 (+ 243us) server_negotiation.cc:407] Connection header received
0927 07:04:18.698524 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:18.698525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:18.698539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:18.698621 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:18.699249 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.699811 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.700545 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:18.700755 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:18.700842 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:18.703146 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:18.703148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:18.703148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:18.703156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:18.704390 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.704748 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.704749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.704749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.704808 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:18.705007 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:18.705008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:18.705008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:18.705162 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:18.705270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:18.705375 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:04:18.705455 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 07:04:18.705883 24568 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46934
I20260927 07:04:18.706400 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24630
I20260927 07:04:18.711503 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24764
I20260927 07:04:18.715996 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24898
I20260927 07:04:18.720232 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 24536
2026-09-27T07:04:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1848 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[25039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[25039](info): set up 2 sockets
Sep 27 07:04:18 dist-test-slave-ltqz krb5kdc[25039](info): commencing operation
krb5kdc: starting...
W20260927 07:04:19.787240 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:19 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:19.906454 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42663
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:19.991745 25052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:19.992002 25052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:19.992039 25052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:19.993216 25052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:19.993294 25052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:19.993319 25052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:19.993350 25052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:19.993382 25052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:19.994911 25052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:42663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:19.995201 25052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:19.995456 25052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:20.000203 25060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.000228 25058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.000250 25057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:20.000392 25052 server_base.cc:1034] running on GCE node
I20260927 07:04:20.000653 25052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:20.001010 25052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:20.002184 25052 hybrid_clock.cc:648] HybridClock initialized: now 1790492660002172 us; error 31 us; skew 500 ppm
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:20.004717 25052 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:20.005105 25052 webserver.cc:460] Webserver started at http://127.21.2.126:35715/ using document root <none> and password file <none>
I20260927 07:04:20.005304 25052 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:20.005345 25052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:20.005434 25052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:20.006445 25052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "39c9d583d19a4c0795663984efb560a9"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "7d29593a7c5945f071ed2ce075f6c212"
server_key_iv: "aeed5aa476ff0a4a4aa789f73ff9d30a"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.006748 25052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "39c9d583d19a4c0795663984efb560a9"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "7d29593a7c5945f071ed2ce075f6c212"
server_key_iv: "aeed5aa476ff0a4a4aa789f73ff9d30a"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.008185 25052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:20.008929 25067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.009128 25052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:20.009222 25052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "39c9d583d19a4c0795663984efb560a9"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "7d29593a7c5945f071ed2ce075f6c212"
server_key_iv: "aeed5aa476ff0a4a4aa789f73ff9d30a"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.009330 25052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:20.034240 25052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:20.036146 25052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:20.036270 25052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:20.040154 25052 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:42663
I20260927 07:04:20.040230 25119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:42663 every 8 connection(s)
I20260927 07:04:20.040707 25052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:20.041412 25120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:20.041661 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25052
I20260927 07:04:20.041756 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:20.041960 21513 external_mini_cluster.cc:1428] Setting key 5703731056736fda5bc706ca5fdce838
I20260927 07:04:20.044750 25120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9: Bootstrap starting.
I20260927 07:04:20.045471 25120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:20.045778 25120 log.cc:824] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:20.046492 25120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9: No bootstrap required, opened a new log
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:20.048581 25120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } }
I20260927 07:04:20.048760 25120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:20.048811 25120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c9d583d19a4c0795663984efb560a9, State: Initialized, Role: FOLLOWER
I20260927 07:04:20.048970 25120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } }
I20260927 07:04:20.049062 25120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:20.049095 25120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:20.049153 25120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:20.049901 25120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } }
I20260927 07:04:20.050010 25120 leader_election.cc:302] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c9d583d19a4c0795663984efb560a9; no voters:
I20260927 07:04:20.050177 25120 leader_election.cc:288] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:20.050283 25125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:20.050462 25125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [term 1 LEADER]: Becoming Leader. State: Replica: 39c9d583d19a4c0795663984efb560a9, State: Running, Role: LEADER
I20260927 07:04:20.050482 25120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:20.050555 25125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } }
I20260927 07:04:20.051110 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c9d583d19a4c0795663984efb560a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } } }
I20260927 07:04:20.051196 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:20.051334 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c9d583d19a4c0795663984efb560a9. Latest consensus state: current_term: 1 leader_uuid: "39c9d583d19a4c0795663984efb560a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9d583d19a4c0795663984efb560a9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42663 } } }
I20260927 07:04:20.051412 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:20.051901 25134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:20.052739 25134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:20.053310 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.042466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.21.2.126:42663)
0927 07:04:20.042633 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.042640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.042665 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:04:20.043102 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.043109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.043158 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.043278 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.044058 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.044974 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.046037 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.046516 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.046649 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.048974 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.048978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.048979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.048992 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.051210 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.051875 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.051878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.051879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.052005 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.052430 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.052431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.052431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.052715 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.052867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.052989 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.053135 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260927 07:04:20.054543 25134 catalog_manager.cc:1347] Generated new cluster ID: 1a6ca3a412f5408399325d78f807a440
I20260927 07:04:20.054597 25134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:20.066351 25134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:20.066886 25134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:20.077275 25134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c9d583d19a4c0795663984efb560a9: Generated new TSK 0
I20260927 07:04:20.077468 25134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:20.106024 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:20.188334 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:20.188541 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:20.188570 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:20.189764 25146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:20.189810 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:20.189837 25146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:20.191500 25146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:20.191740 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:20.191982 25146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:20.196909 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.196949 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.196905 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:20.197257 25146 server_base.cc:1034] running on GCE node
I20260927 07:04:20.197465 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:20.197659 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:20.198802 25146 hybrid_clock.cc:648] HybridClock initialized: now 1790492660198790 us; error 27 us; skew 500 ppm
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:20.201259 25146 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:20.201683 25146 webserver.cc:460] Webserver started at http://127.21.2.65:37903/ using document root <none> and password file <none>
I20260927 07:04:20.201902 25146 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:20.201968 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:20.202124 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:20.203150 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "88b1ed2d917a402bb042dc38f7d7be3a"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "3576c3b772dd92492a70bb13b00ad349"
server_key_iv: "00fd92e7ec2b8e0b31d272ec50a64b1c"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.203545 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "88b1ed2d917a402bb042dc38f7d7be3a"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "3576c3b772dd92492a70bb13b00ad349"
server_key_iv: "00fd92e7ec2b8e0b31d272ec50a64b1c"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.204917 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:20.205715 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.205893 25146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:20.205996 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "88b1ed2d917a402bb042dc38f7d7be3a"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "3576c3b772dd92492a70bb13b00ad349"
server_key_iv: "00fd92e7ec2b8e0b31d272ec50a64b1c"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.206071 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:20.225864 25146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:20.228009 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:20.228176 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:20.228401 25146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:20.228720 25146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:20.228775 25146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.228816 25146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:20.228844 25146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.234500 25146 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:44179
I20260927 07:04:20.234619 25274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:44179 every 8 connection(s)
I20260927 07:04:20.234889 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:20.241047 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25146
I20260927 07:04:20.241171 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:20.241429 21513 external_mini_cluster.cc:1428] Setting key 1f5ce99d58f7b863005a91399a20f963
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:20.246438 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.235665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:34391 (local address 127.21.2.126:42663)
0927 07:04:20.235714 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.235716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.235972 (+ 256us) server_negotiation.cc:407] Connection header received
0927 07:04:20.237396 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.237397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.237413 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.237540 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.238248 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.238816 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.239983 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.240232 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.240338 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.243178 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.243179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.243179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.243186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.244880 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.245384 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.245385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.245385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.245436 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.245716 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.245717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.245717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.245815 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.245864 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.246259 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.246363 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:04:20.246635 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.235642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42663 (local address 127.21.2.65:34391)
0927 07:04:20.235830 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:20.235867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:20.237258 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:20.237625 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:20.237630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:20.237916 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:20.238128 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.238133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.238928 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.238929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.239861 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.239863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.240348 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.240349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.240497 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.241181 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:20.241192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:20.243079 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:20.244975 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.244982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.244984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.245306 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.245500 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.245501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.245501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.245653 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.245880 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:20.245886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:20.246055 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 07:04:20.246180 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:04:20.246367 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260927 07:04:20.247437 25275 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42663
I20260927 07:04:20.247531 25275 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:20.247728 25275 heartbeater.cc:511] Master 127.21.2.126:42663 requested a full tablet report, sending...
I20260927 07:04:20.248304 25084 ts_manager.cc:194] Registered new tserver with Master: 88b1ed2d917a402bb042dc38f7d7be3a (127.21.2.65:44179)
I20260927 07:04:20.249274 25084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:34391
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:20.269481 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:20.357355 25280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:20.357573 25280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:20.357599 25280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:20.358799 25280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:20.358844 25280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:20.358872 25280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:20.360464 25280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:20.360719 25280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:20.360948 25280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:20.365829 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.365850 25285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.365837 25286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:20.366063 25280 server_base.cc:1034] running on GCE node
I20260927 07:04:20.366295 25280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:20.366524 25280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:20.367691 25280 hybrid_clock.cc:648] HybridClock initialized: now 1790492660367679 us; error 36 us; skew 500 ppm
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:20.370175 25280 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:20.370623 25280 webserver.cc:460] Webserver started at http://127.21.2.66:46361/ using document root <none> and password file <none>
I20260927 07:04:20.370818 25280 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:20.370858 25280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:20.370944 25280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:20.372009 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "19b88262d4274f3baeb7a5258c2ce979"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "11baa73894de5c85554bbb3821620f27"
server_key_iv: "05266dfbd291373ca00bbedb4af33638"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.372493 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "19b88262d4274f3baeb7a5258c2ce979"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "11baa73894de5c85554bbb3821620f27"
server_key_iv: "05266dfbd291373ca00bbedb4af33638"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.374393 25280 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:04:20.375381 25295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.375557 25280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:20.375658 25280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "19b88262d4274f3baeb7a5258c2ce979"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "11baa73894de5c85554bbb3821620f27"
server_key_iv: "05266dfbd291373ca00bbedb4af33638"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.375730 25280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:20.388339 25280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:20.390784 25280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:20.390973 25280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:20.391251 25280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:20.391621 25280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:20.391659 25280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.391682 25280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:20.391695 25280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.397540 25280 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:38613
I20260927 07:04:20.397619 25408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:38613 every 8 connection(s)
I20260927 07:04:20.397936 25280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:20.404776 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25280
I20260927 07:04:20.404860 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:20.405043 21513 external_mini_cluster.cc:1428] Setting key 3b908d12bef476af7f6191120b48250d
I20260927 07:04:20.408142 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.398672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:38699 (local address 127.21.2.126:42663)
0927 07:04:20.398742 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.398744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.398949 (+ 205us) server_negotiation.cc:407] Connection header received
0927 07:04:20.399941 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.399942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.399953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.400008 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.400809 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.401297 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.402481 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.402672 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.402728 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.405105 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.405107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.405107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.405122 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.406597 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.407022 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.407024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.407024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.407090 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.407352 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.407353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.407354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.407513 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.407593 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.407951 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.408046 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:04:20.408355 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.398673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42663 (local address 127.21.2.66:38699)
0927 07:04:20.398824 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:20.398860 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:20.399849 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:20.400078 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:20.400081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:20.400527 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:20.400725 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.400730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.401397 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.401398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.402404 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.402407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.402770 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.402771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.402882 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.403482 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:20.403490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:20.405001 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:20.406703 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.406709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.406711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.406933 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.407161 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.407162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.407163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.407279 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.407586 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:20.407592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:20.407739 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 07:04:20.407926 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 07:04:20.408106 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260927 07:04:20.408881 25409 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42663
I20260927 07:04:20.409017 25409 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:20.409229 25409 heartbeater.cc:511] Master 127.21.2.126:42663 requested a full tablet report, sending...
I20260927 07:04:20.409780 25084 ts_manager.cc:194] Registered new tserver with Master: 19b88262d4274f3baeb7a5258c2ce979 (127.21.2.66:38613)
I20260927 07:04:20.410262 25084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:38699
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:20.430591 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:20.513896 25414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:20.514113 25414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:20.514139 25414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:20.515456 25414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:20.515497 25414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:20.515525 25414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:20.517076 25414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:20.517323 25414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:20.517557 25414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:20.522555 25422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.522573 25420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:20.522570 25419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:20.522871 25414 server_base.cc:1034] running on GCE node
I20260927 07:04:20.523113 25414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:20.523355 25414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:20.524569 25414 hybrid_clock.cc:648] HybridClock initialized: now 1790492660524560 us; error 23 us; skew 500 ppm
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:20.526996 25414 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:20.527473 25414 webserver.cc:460] Webserver started at http://127.21.2.67:34391/ using document root <none> and password file <none>
I20260927 07:04:20.527706 25414 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:20.527750 25414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:20.527902 25414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:20.528913 25414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "52b78353ebdd4a12bd0ac1656216e850"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "48a011480623d014f8531bc143e6b403"
server_key_iv: "794c60a174ccb6fd3013e41f60fa6467"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.529277 25414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "52b78353ebdd4a12bd0ac1656216e850"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "48a011480623d014f8531bc143e6b403"
server_key_iv: "794c60a174ccb6fd3013e41f60fa6467"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.530619 25414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:04:20.531522 25429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.531701 25414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:20.531813 25414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "52b78353ebdd4a12bd0ac1656216e850"
format_stamp: "Formatted at 2026-09-27 07:04:20 on dist-test-slave-ltqz"
server_key: "48a011480623d014f8531bc143e6b403"
server_key_iv: "794c60a174ccb6fd3013e41f60fa6467"
server_key_version: "encryptionkey@0"
I20260927 07:04:20.531899 25414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:20.539623 25414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:20.541700 25414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:20.541887 25414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:20.542145 25414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:20.542467 25414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:20.542520 25414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.542560 25414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:20.542605 25414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:20.549099 25414 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:44897
I20260927 07:04:20.549134 25542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:44897 every 8 connection(s)
I20260927 07:04:20.549489 25414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:20.555809 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25414
I20260927 07:04:20.555894 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:20.556085 21513 external_mini_cluster.cc:1428] Setting key 628a3b622c09fa3ed27931eb69cc9e29
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:20.560068 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.550280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:45371 (local address 127.21.2.126:42663)
0927 07:04:20.550336 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.550338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.550561 (+ 223us) server_negotiation.cc:407] Connection header received
0927 07:04:20.551893 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.551895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.551910 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.551943 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.552854 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.553375 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.554402 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.554617 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.554695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.557326 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.557328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.557328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.557336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.558631 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.559063 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.559064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.559065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.559124 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.559397 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.559398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.559398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.559527 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.559605 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.559888 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.559975 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 07:04:20.560350 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.550263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42663 (local address 127.21.2.67:45371)
0927 07:04:20.550437 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:20.550475 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:20.551737 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:20.552116 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:20.552119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:20.552533 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:20.552744 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.552748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.553470 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.553471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.554293 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:20.554296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.554716 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.554717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:20.554841 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.555565 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:20.555574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:20.557226 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:20.558728 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.558736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.558738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.558975 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.559192 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:20.559193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:20.559193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:20.559311 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:20.559592 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:20.559600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:20.559747 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 07:04:20.559851 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:04:20.560018 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260927 07:04:20.560835 25543 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42663
I20260927 07:04:20.560957 25543 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:20.561159 25543 heartbeater.cc:511] Master 127.21.2.126:42663 requested a full tablet report, sending...
I20260927 07:04:20.561560 25084 ts_manager.cc:194] Registered new tserver with Master: 52b78353ebdd4a12bd0ac1656216e850 (127.21.2.67:44897)
I20260927 07:04:20.561944 25084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:45371
I20260927 07:04:20.568106 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:20.591604 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.583815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.21.2.65:44179)
0927 07:04:20.584002 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.584007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.584018 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:04:20.584045 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.584047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.584061 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.584130 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.584689 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.585391 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.586105 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.586465 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.586574 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.588775 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.588783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.588783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.588790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.590407 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.590745 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.590748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.590749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.590815 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.591031 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.591032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.591032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.591198 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.591305 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.591453 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.591524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:20.600219 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:20.592915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.21.2.126:42663)
0927 07:04:20.592988 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:20.592990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:20.592999 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:20.593050 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:20.593051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:20.593061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:20.593108 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:20.593617 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.594122 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.594844 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:20.595042 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:20.595107 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:20.597594 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:20.597596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:20.597596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:20.597604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:20.599007 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.599360 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.599378 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.599379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.599440 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:20.599650 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:20.599651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:20.599652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:20.599815 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:20.599918 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:20.600019 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:04:20.600100 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 07:04:20.600503 25084 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58506
I20260927 07:04:20.600857 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25146
I20260927 07:04:20.605129 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25280
I20260927 07:04:20.609488 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25414
I20260927 07:04:20.613878 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25052
2026-09-27T07:04:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1893 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25555](info): set up 2 sockets
Sep 27 07:04:20 dist-test-slave-ltqz krb5kdc[25555](info): commencing operation
krb5kdc: starting...
W20260927 07:04:21.679291 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:21 dist-test-slave-ltqz krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:21.796484 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:38257
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:21.875824 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:21.876024 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:21.876047 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:21.877159 25568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:21.877218 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:21.877233 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:21.877246 25568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:21.877259 25568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:21.878707 25568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:38257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:21.878908 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:21.879076 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:21.883658 25573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:21.883833 25568 server_base.cc:1034] running on GCE node
W20260927 07:04:21.883909 25576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:21.883913 25574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:21.884276 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:21.884550 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:21.885700 25568 hybrid_clock.cc:648] HybridClock initialized: now 1790492661885685 us; error 29 us; skew 500 ppm
Sep 27 07:04:21 dist-test-slave-ltqz krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492661, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:21.888252 25568 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:21.888549 25568 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:21.888593 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:21.888687 25568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:21.889731 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "f1184f470af846bbad16bab2950139c8"
format_stamp: "Formatted at 2026-09-27 07:04:21 on dist-test-slave-ltqz"
server_key: "7f98432bca54cb8b5776069062c22c09"
server_key_iv: "13d070c0e6225a12242fc363b62633b7"
server_key_version: "encryptionkey@0"
I20260927 07:04:21.890026 25568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "f1184f470af846bbad16bab2950139c8"
format_stamp: "Formatted at 2026-09-27 07:04:21 on dist-test-slave-ltqz"
server_key: "7f98432bca54cb8b5776069062c22c09"
server_key_iv: "13d070c0e6225a12242fc363b62633b7"
server_key_version: "encryptionkey@0"
I20260927 07:04:21.891426 25568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:21.892470 25582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:21.892705 25568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:21.892812 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "f1184f470af846bbad16bab2950139c8"
format_stamp: "Formatted at 2026-09-27 07:04:21 on dist-test-slave-ltqz"
server_key: "7f98432bca54cb8b5776069062c22c09"
server_key_iv: "13d070c0e6225a12242fc363b62633b7"
server_key_version: "encryptionkey@0"
I20260927 07:04:21.892902 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:21.928928 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:21.931036 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:21.931229 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:21.935003 25568 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:38257
I20260927 07:04:21.935031 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:38257 every 8 connection(s)
I20260927 07:04:21.935324 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:21.936281 25635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:21.938570 25635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: Bootstrap starting.
I20260927 07:04:21.939154 25635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:21.939486 25635 log.cc:824] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:21.940155 25635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: No bootstrap required, opened a new log
I20260927 07:04:21.941507 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25568
I20260927 07:04:21.941606 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:21.941807 21513 external_mini_cluster.cc:1428] Setting key 55b26901e07ee1a17d5c2cba48e80623
I20260927 07:04:21.941855 25635 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } }
I20260927 07:04:21.942035 25635 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:21.942075 25635 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1184f470af846bbad16bab2950139c8, State: Initialized, Role: FOLLOWER
I20260927 07:04:21.942242 25635 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } }
I20260927 07:04:21.942376 25635 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:21.942431 25635 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:21.942512 25635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:21.943579 25635 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } }
I20260927 07:04:21.943691 25635 leader_election.cc:302] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1184f470af846bbad16bab2950139c8; no voters:
I20260927 07:04:21.943852 25635 leader_election.cc:288] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:21.943914 25640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:21.944060 25640 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [term 1 LEADER]: Becoming Leader. State: Replica: f1184f470af846bbad16bab2950139c8, State: Running, Role: LEADER
I20260927 07:04:21.944088 25635 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:21.944211 25640 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } }
I20260927 07:04:21.944509 25640 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1184f470af846bbad16bab2950139c8. Latest consensus state: current_term: 1 leader_uuid: "f1184f470af846bbad16bab2950139c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } } }
I20260927 07:04:21.944634 25640 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:21.944517 25642 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1184f470af846bbad16bab2950139c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1184f470af846bbad16bab2950139c8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 38257 } } }
I20260927 07:04:21.944832 25642 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:21.944978 25646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:21.945480 25646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:04:21 dist-test-slave-ltqz krb5kdc[25555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:21.949097 25646 catalog_manager.cc:1347] Generated new cluster ID: 835625457ec3437eba80ffc8f9503353
I20260927 07:04:21.949185 25646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:21.955104 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:21.942310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.21.2.126:38257)
0927 07:04:21.942519 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:21.942530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:21.942562 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:04:21.942675 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:21.942682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:21.942746 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:21.942903 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:21.943527 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:21.945024 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:21.946133 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:21.946613 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:21.946702 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:21.949602 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:21.949606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:21.949607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:21.949621 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:21.952008 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:21.952839 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:21.952847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:21.952848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:21.953547 (+ 699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:21.953852 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:21.953853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:21.953853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:21.954282 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:21.954498 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:21.954643 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 07:04:21.954843 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260927 07:04:21.959076 25646 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:21.959722 25646 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:21.971342 25646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: Generated new TSK 0
I20260927 07:04:21.971594 25646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:21.996402 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:22.078187 25661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:22.078410 25661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:22.078435 25661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:22.079644 25661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:22.079690 25661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:22.079718 25661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:22.081338 25661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:22.081593 25661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:22.081822 25661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:22.086834 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.086841 25661 server_base.cc:1034] running on GCE node
W20260927 07:04:22.086823 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.086807 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.087217 25661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:22.087497 25661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:22.088657 25661 hybrid_clock.cc:648] HybridClock initialized: now 1790492662088649 us; error 37 us; skew 500 ppm
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:22.091194 25661 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:22.091624 25661 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:22.091694 25661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:22.091827 25661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:22.092839 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "14b5b84e8fc844b598c5c6baf491b2ee"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "dcabf6709c2f38d2b3ee066917bb4361"
server_key_iv: "8270e35c402b3ace5a49127ffa2fe263"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.093190 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "14b5b84e8fc844b598c5c6baf491b2ee"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "dcabf6709c2f38d2b3ee066917bb4361"
server_key_iv: "8270e35c402b3ace5a49127ffa2fe263"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.094484 25661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:04:22.095243 25675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.095436 25661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.095530 25661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "14b5b84e8fc844b598c5c6baf491b2ee"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "dcabf6709c2f38d2b3ee066917bb4361"
server_key_iv: "8270e35c402b3ace5a49127ffa2fe263"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.095609 25661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:22.106600 25661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:22.108893 25661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:22.109045 25661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:22.109272 25661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:22.109581 25661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:22.109632 25661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.109673 25661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:22.109699 25661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.115586 25661 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:41057
I20260927 07:04:22.115641 25788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:41057 every 8 connection(s)
I20260927 07:04:22.115926 25661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:22.121284 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25661
I20260927 07:04:22.121410 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:22.121606 21513 external_mini_cluster.cc:1428] Setting key f681dc5ab60512f899c42c433d91694b
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:22.125382 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.116636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:35991 (local address 127.21.2.126:38257)
0927 07:04:22.116720 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.116723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.116910 (+ 187us) server_negotiation.cc:407] Connection header received
0927 07:04:22.117359 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.117360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.117373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.117426 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.118304 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.118785 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.119825 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.120012 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.120064 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.122319 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.122320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.122320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:22.122327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.123791 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.124275 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.124276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.124276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.124326 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.124623 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.124624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.124624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.124771 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.124856 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.125160 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.125264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:04:22.125677 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.116611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:38257 (local address 127.21.2.65:35991)
0927 07:04:22.116787 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.116828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.117274 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.117478 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.117481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.117732 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:22.118228 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.118232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.118861 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.118861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.119747 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.119749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.120078 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.120079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.120172 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.120686 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:22.120694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:22.122199 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:22.123879 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.123886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.123888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.124201 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.124416 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.124417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.124417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.124556 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.124851 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:22.124857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:22.125024 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.125132 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.125303 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260927 07:04:22.126194 25789 heartbeater.cc:347] Connected to a master server at 127.21.2.126:38257
I20260927 07:04:22.126307 25789 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:22.126452 25789 heartbeater.cc:511] Master 127.21.2.126:38257 requested a full tablet report, sending...
I20260927 07:04:22.126977 25599 ts_manager.cc:194] Registered new tserver with Master: 14b5b84e8fc844b598c5c6baf491b2ee (127.21.2.65:41057)
I20260927 07:04:22.127646 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:35991
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:22.146102 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:22.226423 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:22.226660 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:22.226698 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:22.227857 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:22.227929 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:22.227977 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:22.229451 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:22.229705 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:22.229918 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:22.234752 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.234754 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.234791 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.234887 25794 server_base.cc:1034] running on GCE node
I20260927 07:04:22.235090 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:22.235325 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:22.236500 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790492662236482 us; error 36 us; skew 500 ppm
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:22.238838 25794 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:22.239197 25794 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:22.239279 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:22.239432 25794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:22.240523 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "10cdc28d4ac84897b9b193e37ebf4f76"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "32ffc257d6d0b5c64f5d33352aceb486"
server_key_iv: "8395af99c4c0fbafc0117c2a99f45755"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.240882 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "10cdc28d4ac84897b9b193e37ebf4f76"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "32ffc257d6d0b5c64f5d33352aceb486"
server_key_iv: "8395af99c4c0fbafc0117c2a99f45755"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.242168 25794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:22.242872 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.243052 25794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:22.243145 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "10cdc28d4ac84897b9b193e37ebf4f76"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "32ffc257d6d0b5c64f5d33352aceb486"
server_key_iv: "8395af99c4c0fbafc0117c2a99f45755"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.243227 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:22.263028 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:22.265267 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:22.265434 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:22.265669 25794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:22.265987 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:22.266037 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.266078 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:22.266106 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.272003 25794 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:42777
I20260927 07:04:22.272045 25921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:42777 every 8 connection(s)
I20260927 07:04:22.272312 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:22.280998 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25794
I20260927 07:04:22.281132 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:22.281384 21513 external_mini_cluster.cc:1428] Setting key 18d5e87dfcfa9fec6577191f00e49eac
I20260927 07:04:22.281857 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.272999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:42997 (local address 127.21.2.126:38257)
0927 07:04:22.273042 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.273044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.273248 (+ 204us) server_negotiation.cc:407] Connection header received
0927 07:04:22.273695 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.273695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.273704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.273762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.274632 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.275120 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.276188 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.276383 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.276448 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.278655 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.278657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.278657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:22.278667 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.280206 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.280657 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.280659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.280660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.280723 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.281038 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.281039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.281040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.281193 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.281281 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.281654 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.281764 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:04:22.282101 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.272963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:38257 (local address 127.21.2.66:42997)
0927 07:04:22.273112 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.273153 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.273616 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.273812 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.273816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.274060 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:22.274556 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.274560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.275201 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.275201 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.276081 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.276084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.276462 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.276463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.276559 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.277086 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:22.277094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:22.278539 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:22.280314 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.280322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.280324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.280563 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.280793 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.280794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.280794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.280943 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.281291 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:22.281299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:22.281478 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.281629 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.281821 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 07:04:22.282596 25922 heartbeater.cc:347] Connected to a master server at 127.21.2.126:38257
I20260927 07:04:22.282706 25922 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:22.282866 25922 heartbeater.cc:511] Master 127.21.2.126:38257 requested a full tablet report, sending...
I20260927 07:04:22.283303 25599 ts_manager.cc:194] Registered new tserver with Master: 10cdc28d4ac84897b9b193e37ebf4f76 (127.21.2.66:42777)
I20260927 07:04:22.283838 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:42997
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:22.306212 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:22.388207 25927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:22.388432 25927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:22.388473 25927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:22.389675 25927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:22.389732 25927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:22.389778 25927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:22.391297 25927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:22.391577 25927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:22.391798 25927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:22.396661 25935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.396771 25933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.396690 25932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.396931 25927 server_base.cc:1034] running on GCE node
I20260927 07:04:22.397193 25927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:22.397433 25927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:22.398587 25927 hybrid_clock.cc:648] HybridClock initialized: now 1790492662398574 us; error 27 us; skew 500 ppm
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:22.401145 25927 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:22.401470 25927 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:22.401515 25927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:22.401647 25927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:22.402990 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "d233bb359e304965b7a49d59f39a0aba"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "d374f08869a8e175352bb887ddeb37b7"
server_key_iv: "8cb45d701af132aaa28458423d9aa129"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.403337 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "d233bb359e304965b7a49d59f39a0aba"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "d374f08869a8e175352bb887ddeb37b7"
server_key_iv: "8cb45d701af132aaa28458423d9aa129"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.404627 25927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:22.405437 25941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.405617 25927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:22.405718 25927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "d233bb359e304965b7a49d59f39a0aba"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "d374f08869a8e175352bb887ddeb37b7"
server_key_iv: "8cb45d701af132aaa28458423d9aa129"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.405798 25927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:22.418391 25927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:22.420426 25927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:22.420590 25927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:22.420827 25927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:22.421135 25927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:22.421183 25927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.421228 25927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:22.421254 25927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.426805 25927 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:33113
I20260927 07:04:22.426873 26054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:33113 every 8 connection(s)
I20260927 07:04:22.427132 25927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:22.430881 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 25927
I20260927 07:04:22.430986 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:22.431170 21513 external_mini_cluster.cc:1428] Setting key f95edaa24382cb5f1f0192adf7c11d9d
Sep 27 07:04:22 dist-test-slave-ltqz krb5kdc[25555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492662, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:22.436615 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.428006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:46587 (local address 127.21.2.126:38257)
0927 07:04:22.428068 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.428070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.428273 (+ 203us) server_negotiation.cc:407] Connection header received
0927 07:04:22.428717 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.428718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.428730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.428791 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.429665 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.430164 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.431473 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.431743 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.431825 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.433890 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.433892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.433892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:22.433906 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.435510 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.435857 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.435858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.435859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.435913 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.436095 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.436096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.436096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.436221 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.436280 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.436485 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.436546 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:04:22.436798 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.428001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:38257 (local address 127.21.2.67:46587)
0927 07:04:22.428146 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.428179 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.428630 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.428837 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.428841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.429086 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:22.429569 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.429574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.430248 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.430248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.431355 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.431358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.431834 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.431835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.431984 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.432450 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:22.432458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:22.433783 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:22.435576 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.435581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.435582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.435794 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.435964 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:22.435964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:22.435964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:22.436047 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:22.436273 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:22.436275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:22.436381 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.436455 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.436567 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260927 07:04:22.437366 26055 heartbeater.cc:347] Connected to a master server at 127.21.2.126:38257
I20260927 07:04:22.437479 26055 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:22.437640 26055 heartbeater.cc:511] Master 127.21.2.126:38257 requested a full tablet report, sending...
I20260927 07:04:22.438038 25599 ts_manager.cc:194] Registered new tserver with Master: d233bb359e304965b7a49d59f39a0aba (127.21.2.67:33113)
I20260927 07:04:22.438444 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:46587
I20260927 07:04:22.442850 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:04:22.450269 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.444292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.21.2.126:38257)
0927 07:04:22.444352 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.444354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.444494 (+ 140us) server_negotiation.cc:407] Connection header received
0927 07:04:22.444534 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.444535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.444544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.444614 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.445281 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.445865 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.446686 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.446863 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.446919 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.448097 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.448098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.448098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:22.448104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.449453 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.449717 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.449718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.449718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.449764 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:22.449898 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:22.449898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:22.449898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:22.450013 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.450080 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.450158 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.450210 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:04:22.452049 25599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:22.453667 25599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:22.459969 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.455903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:33113 (local address 127.0.0.1:38482)
0927 07:04:22.456119 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.456128 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.456172 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.456581 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.456585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.456587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:22.456652 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.456654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.457582 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.457583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.458728 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.458731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.459630 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.459715 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.459728 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.459818 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.459873 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260927 07:04:22.460096 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.455973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.21.2.65:41057)
0927 07:04:22.456403 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.456407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.456418 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:04:22.456453 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.456455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.456470 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.456539 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:22.456888 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.457984 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459016 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.459718 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459800 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.459853 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.459954 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.460006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":73,"threads_started":1}
I20260927 07:04:22.460268 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.455874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41057 (local address 127.0.0.1:48556)
0927 07:04:22.456054 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.456092 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.456163 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.456765 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.456765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.456767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:22.456803 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.456804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.458084 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.458085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.458938 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.458940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.459820 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.459873 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.459876 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.459926 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.459956 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260927 07:04:22.460430 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.455903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:42777 (local address 127.0.0.1:53132)
0927 07:04:22.456342 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.456350 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.456424 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.456681 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.456682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.456684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:22.456735 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.456737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.458047 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.458048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.459129 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.459131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.460093 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.460094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.460162 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.460167 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.460222 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.460278 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":34,"thread_start_us":51,"threads_started":1}
I20260927 07:04:22.460469 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.456130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.21.2.66:42777)
0927 07:04:22.456313 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.456319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.456422 (+ 103us) server_negotiation.cc:407] Connection header received
0927 07:04:22.456499 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.456501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.456516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.456589 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:22.456831 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.457940 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459207 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.459946 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.460009 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.460057 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.460322 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.460368 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260927 07:04:22.461084 25723 tablet_service.cc:1467] Processing CreateTablet for tablet b7c1239e2caa46a3a371c31eabffe48d (DEFAULT_TABLE table=test-table [id=c1850138417d4c7c8bf4bc1291d755a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:22.461167 25856 tablet_service.cc:1467] Processing CreateTablet for tablet b7c1239e2caa46a3a371c31eabffe48d (DEFAULT_TABLE table=test-table [id=c1850138417d4c7c8bf4bc1291d755a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:22.461447 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c1239e2caa46a3a371c31eabffe48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:22.461503 25856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c1239e2caa46a3a371c31eabffe48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:22.460457 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.456118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.21.2.67:33113)
0927 07:04:22.456315 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.456319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.456330 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:04:22.456372 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.456375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.456391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.456478 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:22.456761 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.457485 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.458827 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.459552 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.459630 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.459690 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.460231 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.460277 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260927 07:04:22.464074 25989 tablet_service.cc:1467] Processing CreateTablet for tablet b7c1239e2caa46a3a371c31eabffe48d (DEFAULT_TABLE table=test-table [id=c1850138417d4c7c8bf4bc1291d755a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:22.464203 26069 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Bootstrap starting.
I20260927 07:04:22.464406 25989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c1239e2caa46a3a371c31eabffe48d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:22.464797 26069 tablet_bootstrap.cc:654] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:22.465307 26069 log.cc:824] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:22.466023 26069 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: No bootstrap required, opened a new log
I20260927 07:04:22.466102 26069 ts_tablet_manager.cc:1397] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:22.466976 26072 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Bootstrap starting.
I20260927 07:04:22.467126 26070 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Bootstrap starting.
I20260927 07:04:22.467736 26070 tablet_bootstrap.cc:654] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:22.467789 26072 tablet_bootstrap.cc:654] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:22.467962 26069 raft_consensus.cc:348] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.468164 26069 raft_consensus.cc:374] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:22.468214 26069 raft_consensus.cc:729] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b5b84e8fc844b598c5c6baf491b2ee, State: Initialized, Role: FOLLOWER
I20260927 07:04:22.468279 26070 log.cc:824] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:22.468374 26072 log.cc:824] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:22.468367 26069 consensus_queue.cc:260] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.468650 26069 ts_tablet_manager.cc:1428] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:22.468811 25789 heartbeater.cc:503] Master 127.21.2.126:38257 was elected leader, sending a full tablet report...
I20260927 07:04:22.469066 26072 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: No bootstrap required, opened a new log
I20260927 07:04:22.469069 26070 tablet_bootstrap.cc:492] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: No bootstrap required, opened a new log
I20260927 07:04:22.469144 26072 ts_tablet_manager.cc:1397] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:04:22.469146 26070 ts_tablet_manager.cc:1397] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:22.470474 26072 raft_consensus.cc:348] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.470650 26072 raft_consensus.cc:374] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:22.470695 26072 raft_consensus.cc:729] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d233bb359e304965b7a49d59f39a0aba, State: Initialized, Role: FOLLOWER
I20260927 07:04:22.470827 26072 consensus_queue.cc:260] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.471119 26072 ts_tablet_manager.cc:1428] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:04:22.470999 26070 raft_consensus.cc:348] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.471140 26055 heartbeater.cc:503] Master 127.21.2.126:38257 was elected leader, sending a full tablet report...
I20260927 07:04:22.471165 26070 raft_consensus.cc:374] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:22.471202 26070 raft_consensus.cc:729] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cdc28d4ac84897b9b193e37ebf4f76, State: Initialized, Role: FOLLOWER
I20260927 07:04:22.471320 26070 consensus_queue.cc:260] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.471624 25922 heartbeater.cc:503] Master 127.21.2.126:38257 was elected leader, sending a full tablet report...
I20260927 07:04:22.471634 26070 ts_tablet_manager.cc:1428] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260927 07:04:22.522840 25923 tablet.cc:2369] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:04:22.616634 25790 tablet.cc:2369] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:22.631132 26077 raft_consensus.cc:482] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:22.631328 26077 raft_consensus.cc:504] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.631740 26077 leader_election.cc:288] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b5b84e8fc844b598c5c6baf491b2ee (127.21.2.65:41057), d233bb359e304965b7a49d59f39a0aba (127.21.2.67:33113)
I20260927 07:04:22.634591 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.632006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:46731 (local address 127.21.2.65:41057)
0927 07:04:22.632054 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.632056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.632073 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:04:22.632162 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.632163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.632175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.632248 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:22.632401 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.632874 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.633866 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.634230 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.634334 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.634344 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.634496 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.634529 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:22.634591 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.631951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41057 (local address 127.21.2.66:46731)
0927 07:04:22.632029 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.632042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.632087 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.632261 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.632262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.632264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:22.632331 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.632333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.632956 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.632957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.633773 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.633776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.634337 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.634338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.634417 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.634420 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.634478 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.634516 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 07:04:22.634650 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.632092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:54009 (local address 127.21.2.67:33113)
0927 07:04:22.632139 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.632141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.632240 (+ 99us) server_negotiation.cc:407] Connection header received
0927 07:04:22.632325 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.632327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.632338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.632390 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:22.632546 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.632996 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.633908 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.634304 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.634369 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.634379 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.634544 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.634577 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:22.634887 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.631951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:33113 (local address 127.21.2.66:54009)
0927 07:04:22.632151 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.632163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.632258 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.632399 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.632399 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.632401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:22.632475 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.632476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.633077 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.633078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.633815 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:22.633817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.634373 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.634373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:22.634460 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.634463 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.634536 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.634575 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":12,"thread_start_us":117,"threads_started":1}
I20260927 07:04:22.634909 25743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c1239e2caa46a3a371c31eabffe48d" candidate_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" is_pre_election: true
I20260927 07:04:22.635077 25743 raft_consensus.cc:2454] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10cdc28d4ac84897b9b193e37ebf4f76 in term 0.
I20260927 07:04:22.635224 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c1239e2caa46a3a371c31eabffe48d" candidate_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233bb359e304965b7a49d59f39a0aba" is_pre_election: true
I20260927 07:04:22.635361 25810 leader_election.cc:302] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10cdc28d4ac84897b9b193e37ebf4f76, 14b5b84e8fc844b598c5c6baf491b2ee; no voters:
I20260927 07:04:22.635408 26009 raft_consensus.cc:2454] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10cdc28d4ac84897b9b193e37ebf4f76 in term 0.
I20260927 07:04:22.635497 26077 raft_consensus.cc:2781] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:22.635555 26077 raft_consensus.cc:482] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:22.635578 26077 raft_consensus.cc:3037] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:22.636247 26077 raft_consensus.cc:504] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.636381 26077 leader_election.cc:288] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [CANDIDATE]: Term 1 election: Requested vote from peers 14b5b84e8fc844b598c5c6baf491b2ee (127.21.2.65:41057), d233bb359e304965b7a49d59f39a0aba (127.21.2.67:33113)
I20260927 07:04:22.636552 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c1239e2caa46a3a371c31eabffe48d" candidate_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d233bb359e304965b7a49d59f39a0aba"
I20260927 07:04:22.636549 25743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7c1239e2caa46a3a371c31eabffe48d" candidate_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b5b84e8fc844b598c5c6baf491b2ee"
I20260927 07:04:22.636619 26009 raft_consensus.cc:3037] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:22.636639 25743 raft_consensus.cc:3037] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:22.637283 26009 raft_consensus.cc:2454] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10cdc28d4ac84897b9b193e37ebf4f76 in term 1.
I20260927 07:04:22.637403 25743 raft_consensus.cc:2454] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10cdc28d4ac84897b9b193e37ebf4f76 in term 1.
I20260927 07:04:22.637452 25809 leader_election.cc:302] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10cdc28d4ac84897b9b193e37ebf4f76, d233bb359e304965b7a49d59f39a0aba; no voters:
I20260927 07:04:22.637526 26077 raft_consensus.cc:2781] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:22.637689 26077 raft_consensus.cc:686] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 1 LEADER]: Becoming Leader. State: Replica: 10cdc28d4ac84897b9b193e37ebf4f76, State: Running, Role: LEADER
I20260927 07:04:22.637800 26077 consensus_queue.cc:237] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } }
I20260927 07:04:22.638597 25598 catalog_manager.cc:5495] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10cdc28d4ac84897b9b193e37ebf4f76 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10cdc28d4ac84897b9b193e37ebf4f76" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 42777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:22.643497 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.641455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.21.2.66:42777)
0927 07:04:22.641492 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.641494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.641503 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:22.641557 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.641558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.641573 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.641636 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:22.641731 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.642152 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.642839 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:22.643007 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:22.643054 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:22.643180 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:22.643261 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:22.643345 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.643418 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.643435 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:04:22.645176 25743 raft_consensus.cc:1261] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 1 FOLLOWER]: Refusing update from remote peer 10cdc28d4ac84897b9b193e37ebf4f76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:22.645258 26009 raft_consensus.cc:1261] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 1 FOLLOWER]: Refusing update from remote peer 10cdc28d4ac84897b9b193e37ebf4f76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:22.645496 26077 consensus_queue.cc:1035] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b5b84e8fc844b598c5c6baf491b2ee" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:22.645624 26079 consensus_queue.cc:1035] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d233bb359e304965b7a49d59f39a0aba" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 33113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:22.649573 25598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:22.649670 25598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:22.650704 25598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1184f470af846bbad16bab2950139c8: Sending DeleteTablet for 3 replicas of tablet b7c1239e2caa46a3a371c31eabffe48d
I20260927 07:04:22.651083 25856 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c1239e2caa46a3a371c31eabffe48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:22 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 07:04:22.651094 25989 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c1239e2caa46a3a371c31eabffe48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:22 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:38482
I20260927 07:04:22.651319 26092 tablet_replica.cc:330] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76: stopping tablet replica
I20260927 07:04:22.651343 25723 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c1239e2caa46a3a371c31eabffe48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:22 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48556
I20260927 07:04:22.651449 26092 raft_consensus.cc:2229] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:22.651543 26092 raft_consensus.cc:2258] T b7c1239e2caa46a3a371c31eabffe48d P 10cdc28d4ac84897b9b193e37ebf4f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:22.651585 25722 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c1239e2caa46a3a371c31eabffe48d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:22 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48556
I20260927 07:04:22.651679 26094 tablet_replica.cc:330] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee: stopping tablet replica
I20260927 07:04:22.651811 26094 raft_consensus.cc:2229] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:22.651970 26094 raft_consensus.cc:2258] T b7c1239e2caa46a3a371c31eabffe48d P 14b5b84e8fc844b598c5c6baf491b2ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:22.652137 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25661
W20260927 07:04:22.658007 25584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:41057 (error 108)
I20260927 07:04:22.658227 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25794
I20260927 07:04:22.658361 26093 tablet_replica.cc:330] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: stopping tablet replica
I20260927 07:04:22.658470 26093 raft_consensus.cc:2229] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:22.658562 26093 raft_consensus.cc:2258] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:22.659021 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.658887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41057 (local address 127.0.0.1:48566)
0927 07:04:22.658928 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.658979 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 07:04:22.659204 25584 catalog_manager.cc:4584] TS 14b5b84e8fc844b598c5c6baf491b2ee (127.21.2.65:41057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7c1239e2caa46a3a371c31eabffe48d: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41057: connect: Connection refused (error 111)
I20260927 07:04:22.662528 26093 ts_tablet_manager.cc:1905] T b7c1239e2caa46a3a371c31eabffe48d P d233bb359e304965b7a49d59f39a0aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:22.662724 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25927
I20260927 07:04:22.662798 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.662691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:42777 (local address 127.0.0.1:53140)
0927 07:04:22.662727 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.662757 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.66:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 07:04:22.666886 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 25568
2026-09-27T07:04:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2053 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T07:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:22Z Disabled control of system clock
I20260927 07:04:22.694519 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40395
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:04:22.776419 26099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:22.776621 26099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:22.776643 26099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:22.777712 26099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:22.777746 26099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:22.777758 26099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:22.777771 26099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:22.777783 26099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:22.779129 26099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:22.779338 26099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:22.779616 26099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:22.784132 26105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.784179 26107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.784359 26104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.784597 26099 server_base.cc:1034] running on GCE node
I20260927 07:04:22.784847 26099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:22.785133 26099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:22.786315 26099 hybrid_clock.cc:648] HybridClock initialized: now 1790492662786294 us; error 37 us; skew 500 ppm
I20260927 07:04:22.787622 26099 webserver.cc:460] Webserver started at http://127.21.2.126:38943/ using document root <none> and password file <none>
I20260927 07:04:22.787868 26099 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:22.787935 26099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:22.788080 26099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:22.789273 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "1d71206bc9fd47fcaf1ac5b08484f20f"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "320278430a82685c5af195d240241843"
server_key_iv: "697d7b28fb6f3f39f2e459467b5d52bd"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.789664 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "1d71206bc9fd47fcaf1ac5b08484f20f"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "320278430a82685c5af195d240241843"
server_key_iv: "697d7b28fb6f3f39f2e459467b5d52bd"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.790974 26099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:22.791775 26113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.791939 26099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.792040 26099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "1d71206bc9fd47fcaf1ac5b08484f20f"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "320278430a82685c5af195d240241843"
server_key_iv: "697d7b28fb6f3f39f2e459467b5d52bd"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.792111 26099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:22.795428 26099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:22.795699 26099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:22.795832 26099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:22.799750 26099 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40395
I20260927 07:04:22.799952 26165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40395 every 8 connection(s)
I20260927 07:04:22.800362 26099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:22.801309 26166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:22.804724 26166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: Bootstrap starting.
I20260927 07:04:22.805279 26166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:22.805521 26166 log.cc:824] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:22.806522 26166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: No bootstrap required, opened a new log
I20260927 07:04:22.808598 26166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } }
I20260927 07:04:22.808845 26166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:22.808890 26166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d71206bc9fd47fcaf1ac5b08484f20f, State: Initialized, Role: FOLLOWER
I20260927 07:04:22.808887 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26099
I20260927 07:04:22.808979 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:22.809051 26166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } }
I20260927 07:04:22.809152 26166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:22.809170 21513 external_mini_cluster.cc:1428] Setting key 1828526920a8427670dbbff86a0e3269
I20260927 07:04:22.809185 26166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:22.809233 26166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:22.810019 26166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } }
I20260927 07:04:22.810153 26166 leader_election.cc:302] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d71206bc9fd47fcaf1ac5b08484f20f; no voters:
I20260927 07:04:22.810343 26166 leader_election.cc:288] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:22.810442 26171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:22.810892 26171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [term 1 LEADER]: Becoming Leader. State: Replica: 1d71206bc9fd47fcaf1ac5b08484f20f, State: Running, Role: LEADER
I20260927 07:04:22.810986 26171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } }
I20260927 07:04:22.810994 26166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:22.811571 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.809683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.21.2.126:40395)
0927 07:04:22.809901 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.809908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.809934 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:04:22.810026 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.810032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.810065 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.810219 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.810899 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.810903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.810903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:22.810909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.810952 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:22.810966 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.811023 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.811273 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.811332 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260927 07:04:22.811560 26172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } } }
I20260927 07:04:22.811676 26172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:22.812412 26180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:22.812441 26173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d71206bc9fd47fcaf1ac5b08484f20f. Latest consensus state: current_term: 1 leader_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d71206bc9fd47fcaf1ac5b08484f20f" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40395 } } }
I20260927 07:04:22.812724 26173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:22.812944 26180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:22.814275 26180 catalog_manager.cc:1347] Generated new cluster ID: dcee6c7a70164e50a16a9752c8025566
I20260927 07:04:22.814328 26180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:22.847723 26180 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:22.847930 26180 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:22.854219 26180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: Generated new TSK 0
I20260927 07:04:22.854430 26180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:04:22.860998 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--builtin_ntp_servers=127.21.2.84:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:04:22.950800 26190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:22.951026 26190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:22.951049 26190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:22.952675 26190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:22.952737 26190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:22.952765 26190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:22.954316 26190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:22.954597 26190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:22.954909 26190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:22.959622 26195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.959656 26198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:22.959645 26196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:22.959867 26190 server_base.cc:1034] running on GCE node
I20260927 07:04:22.960076 26190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:22.960323 26190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:22.961480 26190 hybrid_clock.cc:648] HybridClock initialized: now 1790492662961468 us; error 28 us; skew 500 ppm
I20260927 07:04:22.962646 26190 webserver.cc:460] Webserver started at http://127.21.2.65:42813/ using document root <none> and password file <none>
I20260927 07:04:22.962946 26190 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:22.963013 26190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:22.963147 26190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:22.964267 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "06fdeff4b9c34a9c89ae7d24f11203d2"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "07dd1a2851e32a9a37f21291d107d46b"
server_key_iv: "3584c9a94c913ef70b075deba6e1908f"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.964618 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "06fdeff4b9c34a9c89ae7d24f11203d2"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "07dd1a2851e32a9a37f21291d107d46b"
server_key_iv: "3584c9a94c913ef70b075deba6e1908f"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.965883 26190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:22.966583 26204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.966738 26190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:22.966837 26190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "06fdeff4b9c34a9c89ae7d24f11203d2"
format_stamp: "Formatted at 2026-09-27 07:04:22 on dist-test-slave-ltqz"
server_key: "07dd1a2851e32a9a37f21291d107d46b"
server_key_iv: "3584c9a94c913ef70b075deba6e1908f"
server_key_version: "encryptionkey@0"
I20260927 07:04:22.966920 26190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:22.970186 26190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:22.970440 26190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:22.970582 26190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:22.970786 26190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:22.971092 26190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:22.971143 26190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.971194 26190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:22.971220 26190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:22.977967 26317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:44331 every 8 connection(s)
I20260927 07:04:22.977972 26190 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:44331
I20260927 07:04:22.978478 26190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:22.981607 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.979191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:48611 (local address 127.21.2.126:40395)
0927 07:04:22.979257 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:22.979260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:22.979533 (+ 273us) server_negotiation.cc:407] Connection header received
0927 07:04:22.980037 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:22.980039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:22.980047 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:22.980105 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:22.981354 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:22.981355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:22.981355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:22.981360 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:22.981376 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:22.981381 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:22.981431 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:22.981517 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:04:22.981533 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:04:22.981806 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:22.979195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40395 (local address 127.21.2.65:48611)
0927 07:04:22.979410 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:22.979450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:22.979931 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:22.980177 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:22.980185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:22.980193 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:22.981208 (+ 1015us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:22.981208 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:22.981235 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:22.981261 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:22.981433 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:22.981437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:22.981440 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:22.981528 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 07:04:22.981587 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260927 07:04:22.982365 26318 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40395
I20260927 07:04:22.982467 26318 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:22.982646 26318 heartbeater.cc:511] Master 127.21.2.126:40395 requested a full tablet report, sending...
I20260927 07:04:22.983086 26130 ts_manager.cc:194] Registered new tserver with Master: 06fdeff4b9c34a9c89ae7d24f11203d2 (127.21.2.65:44331)
I20260927 07:04:22.986377 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26190
I20260927 07:04:22.986469 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:22.986603 21513 external_mini_cluster.cc:1428] Setting key 2df730027bc900b01dd838bbfb2dfe41
I20260927 07:04:22.987551 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--builtin_ntp_servers=127.21.2.84:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:04:23.068187 26321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:23.068385 26321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:23.068408 26321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:23.069554 26321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:23.069593 26321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:23.069628 26321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:23.070946 26321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:23.071121 26321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:23.071352 26321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:23.076148 26326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.076152 26329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.076155 26327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.076359 26321 server_base.cc:1034] running on GCE node
I20260927 07:04:23.076567 26321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:23.076830 26321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:23.077970 26321 hybrid_clock.cc:648] HybridClock initialized: now 1790492663077957 us; error 27 us; skew 500 ppm
I20260927 07:04:23.079170 26321 webserver.cc:460] Webserver started at http://127.21.2.66:41449/ using document root <none> and password file <none>
I20260927 07:04:23.079452 26321 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:23.079516 26321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:23.079668 26321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:23.080690 26321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "0f2c26959f3d47dfa89359a889d61d30"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "1dc89c7f6ccc5dd8d60522e306dde3fb"
server_key_iv: "8db67bfb0cfb6e5a2be1f09a96168fb5"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.081053 26321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "0f2c26959f3d47dfa89359a889d61d30"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "1dc89c7f6ccc5dd8d60522e306dde3fb"
server_key_iv: "8db67bfb0cfb6e5a2be1f09a96168fb5"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.082329 26321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:23.083001 26335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.083159 26321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:23.083251 26321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "0f2c26959f3d47dfa89359a889d61d30"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "1dc89c7f6ccc5dd8d60522e306dde3fb"
server_key_iv: "8db67bfb0cfb6e5a2be1f09a96168fb5"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.083333 26321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:23.086602 26321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:23.086858 26321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:23.086997 26321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:23.087204 26321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:23.087556 26321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:23.087605 26321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.087644 26321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:23.087670 26321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.093233 26321 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:43035
I20260927 07:04:23.093305 26448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:43035 every 8 connection(s)
I20260927 07:04:23.093580 26321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:23.096331 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.094213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:45191 (local address 127.21.2.126:40395)
0927 07:04:23.094298 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.094301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.094516 (+ 215us) server_negotiation.cc:407] Connection header received
0927 07:04:23.094981 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.094982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.094992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.095034 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.096121 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.096122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.096122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.096126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.096140 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.096144 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.096197 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.096258 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.096270 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:04:23.096498 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.094211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40395 (local address 127.21.2.66:45191)
0927 07:04:23.094406 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.094438 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.094895 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.095084 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.095087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.095094 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.095994 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.095994 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.096020 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.096046 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.096197 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.096199 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.096201 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.096263 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.096308 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260927 07:04:23.097012 26449 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40395
I20260927 07:04:23.097112 26449 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:23.097249 26449 heartbeater.cc:511] Master 127.21.2.126:40395 requested a full tablet report, sending...
I20260927 07:04:23.097615 26130 ts_manager.cc:194] Registered new tserver with Master: 0f2c26959f3d47dfa89359a889d61d30 (127.21.2.66:43035)
I20260927 07:04:23.102118 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26321
I20260927 07:04:23.102200 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:23.102378 21513 external_mini_cluster.cc:1428] Setting key 37e2b65546e677f2fc2f08c92cf7c9d1
I20260927 07:04:23.103282 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--builtin_ntp_servers=127.21.2.84:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:04:23.184310 26452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:23.184499 26452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:23.184520 26452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:23.185689 26452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:23.185731 26452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:23.185757 26452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:23.187108 26452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:23.187362 26452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:23.187646 26452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:23.192389 26457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.192448 26458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.192395 26460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.192500 26452 server_base.cc:1034] running on GCE node
I20260927 07:04:23.192785 26452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:23.193014 26452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:23.194185 26452 hybrid_clock.cc:648] HybridClock initialized: now 1790492663194174 us; error 34 us; skew 500 ppm
I20260927 07:04:23.195571 26452 webserver.cc:460] Webserver started at http://127.21.2.67:41715/ using document root <none> and password file <none>
I20260927 07:04:23.195819 26452 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:23.195864 26452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:23.196024 26452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:23.197122 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "5a758e73e4554737ab9948bd459d4947"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "e25b9fcfb92204117a497c46ae7a942f"
server_key_iv: "9588dc793270eed08d32ac4715319ebd"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.197474 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "5a758e73e4554737ab9948bd459d4947"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "e25b9fcfb92204117a497c46ae7a942f"
server_key_iv: "9588dc793270eed08d32ac4715319ebd"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.198777 26452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:23.199530 26466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.199700 26452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:23.199785 26452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "5a758e73e4554737ab9948bd459d4947"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "e25b9fcfb92204117a497c46ae7a942f"
server_key_iv: "9588dc793270eed08d32ac4715319ebd"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.199857 26452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:23.203115 26452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:23.203409 26452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:23.203557 26452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:23.203765 26452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:23.204052 26452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:23.204083 26452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.204133 26452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:23.204161 26452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.209996 26452 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:45519
I20260927 07:04:23.210040 26579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:45519 every 8 connection(s)
I20260927 07:04:23.210422 26452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:23.213726 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.211263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:59003 (local address 127.21.2.126:40395)
0927 07:04:23.211303 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.211305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.211636 (+ 331us) server_negotiation.cc:407] Connection header received
0927 07:04:23.212200 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.212202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.212208 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.212271 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.213454 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.213456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.213468 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.213473 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.213487 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.213491 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.213558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.213638 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.213655 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:23.213922 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.211260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40395 (local address 127.21.2.67:59003)
0927 07:04:23.211499 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.211542 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.212110 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.212334 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.212339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.212348 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.213302 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.213303 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.213329 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.213355 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.213555 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.213561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.213564 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.213632 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.213693 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260927 07:04:23.214394 26580 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40395
I20260927 07:04:23.214493 26580 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:23.214636 26580 heartbeater.cc:511] Master 127.21.2.126:40395 requested a full tablet report, sending...
I20260927 07:04:23.215004 26130 ts_manager.cc:194] Registered new tserver with Master: 5a758e73e4554737ab9948bd459d4947 (127.21.2.67:45519)
I20260927 07:04:23.217803 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26452
I20260927 07:04:23.217904 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:23.218101 21513 external_mini_cluster.cc:1428] Setting key c871b5e593082e3b5063566c8450be05
I20260927 07:04:23.219043 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:04:23.221346 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.220452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.21.2.126:40395)
0927 07:04:23.220490 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.220492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.220591 (+ 99us) server_negotiation.cc:407] Connection header received
0927 07:04:23.220645 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.220645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.220649 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.220695 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.221180 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.221181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.221181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.221185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.221199 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.221203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.221255 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.221288 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.221301 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:04:23.222740 26130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:23.224339 26130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:23.228009 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:43035 (local address 127.0.0.1:48646)
0927 07:04:23.226757 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.226766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.226829 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.227012 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.227012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.227015 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.227531 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.227532 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.227549 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.227573 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.227819 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.227825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.227827 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.227908 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.227922 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":82,"threads_started":1}
I20260927 07:04:23.228039 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.21.2.66:43035)
0927 07:04:23.226651 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.226656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.226824 (+ 168us) server_negotiation.cc:407] Connection header received
0927 07:04:23.226892 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.226895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.226906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.227211 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.227668 (+ 457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.227675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.227675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.227680 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.227727 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.227757 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.227825 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.227920 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.227937 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260927 07:04:23.228327 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.21.2.67:45519)
0927 07:04:23.226700 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.226704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.226976 (+ 272us) server_negotiation.cc:407] Connection header received
0927 07:04:23.227068 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.227072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.227081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.227132 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.228003 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.228009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.228009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.228013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.228046 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.228070 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.228112 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.228228 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.228246 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260927 07:04:23.228273 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45519 (local address 127.0.0.1:54162)
0927 07:04:23.226918 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.226929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.226996 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.227486 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.227486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.227488 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.227894 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.227895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.227902 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.227915 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.228162 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.228164 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.228164 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.228212 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.228223 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":71,"threads_started":1}
I20260927 07:04:23.228561 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44331 (local address 127.0.0.1:43132)
0927 07:04:23.226650 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.226691 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.226748 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.226930 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.226935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.226940 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.228167 (+ 1227us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.228168 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.228175 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.228186 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.228445 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.228446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.228446 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.228490 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.228504 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":69,"threads_started":1}
I20260927 07:04:23.228793 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.226568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.21.2.65:44331)
0927 07:04:23.226795 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.226799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.226808 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:23.226831 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.226833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.226841 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.226930 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.228285 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.228293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.228293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.228297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.228331 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.228360 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.228419 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.228696 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.228713 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260927 07:04:23.229007 26514 tablet_service.cc:1467] Processing CreateTablet for tablet 440051bce12e4bb8acca0a1946f99cce (DEFAULT_TABLE table=test-table [id=1df20cde2b0541ca8ff539b310daa211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:23.229099 26383 tablet_service.cc:1467] Processing CreateTablet for tablet 440051bce12e4bb8acca0a1946f99cce (DEFAULT_TABLE table=test-table [id=1df20cde2b0541ca8ff539b310daa211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:23.229362 26514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440051bce12e4bb8acca0a1946f99cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:23.229450 26383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440051bce12e4bb8acca0a1946f99cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:23.229406 26252 tablet_service.cc:1467] Processing CreateTablet for tablet 440051bce12e4bb8acca0a1946f99cce (DEFAULT_TABLE table=test-table [id=1df20cde2b0541ca8ff539b310daa211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:23.229699 26252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 440051bce12e4bb8acca0a1946f99cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:23.232015 26594 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Bootstrap starting.
I20260927 07:04:23.232640 26595 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Bootstrap starting.
I20260927 07:04:23.232750 26594 tablet_bootstrap.cc:654] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:23.232849 26596 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Bootstrap starting.
I20260927 07:04:23.233183 26594 log.cc:824] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:23.233299 26595 tablet_bootstrap.cc:654] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:23.233455 26596 tablet_bootstrap.cc:654] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:23.233613 26595 log.cc:824] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:23.233783 26596 log.cc:824] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:23.234150 26594 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: No bootstrap required, opened a new log
I20260927 07:04:23.234236 26594 ts_tablet_manager.cc:1397] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:23.234467 26595 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: No bootstrap required, opened a new log
I20260927 07:04:23.234508 26595 ts_tablet_manager.cc:1397] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:23.235780 26596 tablet_bootstrap.cc:492] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: No bootstrap required, opened a new log
I20260927 07:04:23.235884 26596 ts_tablet_manager.cc:1397] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:04:23.236182 26594 raft_consensus.cc:348] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.236338 26594 raft_consensus.cc:374] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:23.236382 26594 raft_consensus.cc:729] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a758e73e4554737ab9948bd459d4947, State: Initialized, Role: FOLLOWER
I20260927 07:04:23.236513 26594 consensus_queue.cc:260] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.236792 26594 ts_tablet_manager.cc:1428] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:23.236703 26595 raft_consensus.cc:348] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.236852 26595 raft_consensus.cc:374] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:23.236896 26595 raft_consensus.cc:729] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2c26959f3d47dfa89359a889d61d30, State: Initialized, Role: FOLLOWER
I20260927 07:04:23.236904 26580 heartbeater.cc:503] Master 127.21.2.126:40395 was elected leader, sending a full tablet report...
I20260927 07:04:23.236999 26595 consensus_queue.cc:260] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.237236 26595 ts_tablet_manager.cc:1428] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:23.237617 26449 heartbeater.cc:503] Master 127.21.2.126:40395 was elected leader, sending a full tablet report...
I20260927 07:04:23.238566 26596 raft_consensus.cc:348] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.238694 26596 raft_consensus.cc:374] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:23.238716 26596 raft_consensus.cc:729] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fdeff4b9c34a9c89ae7d24f11203d2, State: Initialized, Role: FOLLOWER
I20260927 07:04:23.238796 26596 consensus_queue.cc:260] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.239001 26596 ts_tablet_manager.cc:1428] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:04:23.239035 26318 heartbeater.cc:503] Master 127.21.2.126:40395 was elected leader, sending a full tablet report...
W20260927 07:04:23.344110 26450 tablet.cc:2369] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:04:23.460991 26581 tablet.cc:2369] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:23.478186 26601 raft_consensus.cc:482] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:23.478329 26601 raft_consensus.cc:504] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.478682 26601 leader_election.cc:288] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06fdeff4b9c34a9c89ae7d24f11203d2 (127.21.2.65:44331), 5a758e73e4554737ab9948bd459d4947 (127.21.2.67:45519)
W20260927 07:04:23.479303 26319 tablet.cc:2369] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:04:23.480038 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.478901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44331 (local address 127.21.2.66:53349)
0927 07:04:23.478961 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.478974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.479014 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.479180 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.479182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.479184 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.479678 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.479678 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.479685 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.479696 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.479890 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.479891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.479892 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.479941 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.479954 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 07:04:23.480070 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.479002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:53349 (local address 127.21.2.65:44331)
0927 07:04:23.479049 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.479051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.479067 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:04:23.479101 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.479102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.479107 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.479174 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.479793 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.479795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.479795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.479800 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.479814 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.479818 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.479881 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.479966 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.479981 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:23.480147 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.478970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:44825 (local address 127.21.2.67:45519)
0927 07:04:23.479033 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.479035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.479239 (+ 204us) server_negotiation.cc:407] Connection header received
0927 07:04:23.479299 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.479300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.479306 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.479377 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.479930 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.479931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.479932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.479936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.479953 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.479958 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.480010 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.480055 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.480067 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:04:23.480152 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.478917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45519 (local address 127.21.2.66:44825)
0927 07:04:23.479113 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.479128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.479244 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.479388 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.479388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.479390 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:23.479855 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:04:23.479856 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:23.479862 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:04:23.479873 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:23.480008 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:23.480009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:23.480010 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:04:23.480057 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:04:23.480068 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260927 07:04:23.480427 26272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440051bce12e4bb8acca0a1946f99cce" candidate_uuid: "0f2c26959f3d47dfa89359a889d61d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" is_pre_election: true
I20260927 07:04:23.480456 26534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440051bce12e4bb8acca0a1946f99cce" candidate_uuid: "0f2c26959f3d47dfa89359a889d61d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a758e73e4554737ab9948bd459d4947" is_pre_election: true
I20260927 07:04:23.480575 26534 raft_consensus.cc:2454] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2c26959f3d47dfa89359a889d61d30 in term 0.
I20260927 07:04:23.480573 26272 raft_consensus.cc:2454] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2c26959f3d47dfa89359a889d61d30 in term 0.
I20260927 07:04:23.480805 26336 leader_election.cc:302] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2c26959f3d47dfa89359a889d61d30, 5a758e73e4554737ab9948bd459d4947; no voters:
I20260927 07:04:23.480930 26601 raft_consensus.cc:2781] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:23.480981 26601 raft_consensus.cc:482] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:23.481010 26601 raft_consensus.cc:3037] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:23.481668 26601 raft_consensus.cc:504] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.481804 26601 leader_election.cc:288] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [CANDIDATE]: Term 1 election: Requested vote from peers 06fdeff4b9c34a9c89ae7d24f11203d2 (127.21.2.65:44331), 5a758e73e4554737ab9948bd459d4947 (127.21.2.67:45519)
I20260927 07:04:23.481941 26534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440051bce12e4bb8acca0a1946f99cce" candidate_uuid: "0f2c26959f3d47dfa89359a889d61d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a758e73e4554737ab9948bd459d4947"
I20260927 07:04:23.481938 26272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "440051bce12e4bb8acca0a1946f99cce" candidate_uuid: "0f2c26959f3d47dfa89359a889d61d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2"
I20260927 07:04:23.482018 26534 raft_consensus.cc:3037] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:23.482026 26272 raft_consensus.cc:3037] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:23.482733 26534 raft_consensus.cc:2454] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2c26959f3d47dfa89359a889d61d30 in term 1.
I20260927 07:04:23.482739 26272 raft_consensus.cc:2454] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2c26959f3d47dfa89359a889d61d30 in term 1.
I20260927 07:04:23.482923 26336 leader_election.cc:302] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2c26959f3d47dfa89359a889d61d30, 5a758e73e4554737ab9948bd459d4947; no voters:
I20260927 07:04:23.483026 26601 raft_consensus.cc:2781] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:23.483155 26601 raft_consensus.cc:686] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2c26959f3d47dfa89359a889d61d30, State: Running, Role: LEADER
I20260927 07:04:23.483246 26601 consensus_queue.cc:237] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } }
I20260927 07:04:23.484045 26130 catalog_manager.cc:5495] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2c26959f3d47dfa89359a889d61d30 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "0f2c26959f3d47dfa89359a889d61d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2c26959f3d47dfa89359a889d61d30" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:23.513315 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.512348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.21.2.66:43035)
0927 07:04:23.512393 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.512395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.512404 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:23.512471 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.512472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.512483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.512531 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.513123 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.513124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.513125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.513128 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.513142 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.513145 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.513192 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.513246 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.513257 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:23.515120 26272 raft_consensus.cc:1261] T 440051bce12e4bb8acca0a1946f99cce P 06fdeff4b9c34a9c89ae7d24f11203d2 [term 1 FOLLOWER]: Refusing update from remote peer 0f2c26959f3d47dfa89359a889d61d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:23.515120 26534 raft_consensus.cc:1261] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 1 FOLLOWER]: Refusing update from remote peer 0f2c26959f3d47dfa89359a889d61d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:23.515467 26604 consensus_queue.cc:1035] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a758e73e4554737ab9948bd459d4947" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:23.515641 26601 consensus_queue.cc:1035] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06fdeff4b9c34a9c89ae7d24f11203d2" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:23.518463 26608 mvcc.cc:205] Tried to move back new op lower bound from 7333857949754916864 to 7333857949627990016. Current Snapshot: MvccSnapshot[applied={T|T < 7333857949754916864}]
I20260927 07:04:23.518536 26609 mvcc.cc:205] Tried to move back new op lower bound from 7333857949754916864 to 7333857949627990016. Current Snapshot: MvccSnapshot[applied={T|T < 7333857949754916864}]
I20260927 07:04:23.520246 26130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:23.520335 26130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:23.521071 26130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d71206bc9fd47fcaf1ac5b08484f20f: Sending DeleteTablet for 3 replicas of tablet 440051bce12e4bb8acca0a1946f99cce
I20260927 07:04:23.521409 26514 tablet_service.cc:1514] Processing DeleteTablet for tablet 440051bce12e4bb8acca0a1946f99cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:23 UTC) from {username='slave'} at 127.0.0.1:54162
I20260927 07:04:23.521427 26252 tablet_service.cc:1514] Processing DeleteTablet for tablet 440051bce12e4bb8acca0a1946f99cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:23 UTC) from {username='slave'} at 127.0.0.1:43132
I20260927 07:04:23.521688 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26190
I20260927 07:04:23.521699 26383 tablet_service.cc:1514] Processing DeleteTablet for tablet 440051bce12e4bb8acca0a1946f99cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:23 UTC) from {username='slave'} at 127.0.0.1:48646
I20260927 07:04:23.522392 26618 tablet_replica.cc:330] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: stopping tablet replica
I20260927 07:04:23.522553 26618 raft_consensus.cc:2229] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:23.522701 26618 raft_consensus.cc:2258] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:23.523219 26618 ts_tablet_manager.cc:1905] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:23.523603 26616 tablet_replica.cc:330] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: stopping tablet replica
I20260927 07:04:23.523713 26616 raft_consensus.cc:2229] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:23.523826 26616 raft_consensus.cc:2258] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:23.524241 26616 ts_tablet_manager.cc:1905] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:23.525846 26616 ts_tablet_manager.cc:1918] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:23.525920 26616 log.cc:1196] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/440051bce12e4bb8acca0a1946f99cce
I20260927 07:04:23.526091 26618 ts_tablet_manager.cc:1918] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:23.526180 26616 ts_tablet_manager.cc:1939] T 440051bce12e4bb8acca0a1946f99cce P 5a758e73e4554737ab9948bd459d4947: Deleting consensus metadata
I20260927 07:04:23.526285 26618 log.cc:1196] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/440051bce12e4bb8acca0a1946f99cce
I20260927 07:04:23.526574 26114 catalog_manager.cc:4841] TS 5a758e73e4554737ab9948bd459d4947 (127.21.2.67:45519): tablet 440051bce12e4bb8acca0a1946f99cce (table test-table [id=1df20cde2b0541ca8ff539b310daa211]) successfully deleted
I20260927 07:04:23.526696 26618 ts_tablet_manager.cc:1939] T 440051bce12e4bb8acca0a1946f99cce P 0f2c26959f3d47dfa89359a889d61d30: Deleting consensus metadata
I20260927 07:04:23.527140 26115 catalog_manager.cc:4841] TS 0f2c26959f3d47dfa89359a889d61d30 (127.21.2.66:43035): tablet 440051bce12e4bb8acca0a1946f99cce (table test-table [id=1df20cde2b0541ca8ff539b310daa211]) successfully deleted
W20260927 07:04:23.528259 26116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:44331 (error 108)
I20260927 07:04:23.528530 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26321
I20260927 07:04:23.529182 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.528610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44331 (local address 127.0.0.1:43146)
0927 07:04:23.529084 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.529140 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
W20260927 07:04:23.529632 26116 catalog_manager.cc:4584] TS 06fdeff4b9c34a9c89ae7d24f11203d2 (127.21.2.65:44331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 440051bce12e4bb8acca0a1946f99cce: Network error: Client connection negotiation failed: client connection to 127.21.2.65:44331: connect: Connection refused (error 111)
I20260927 07:04:23.533566 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26452
I20260927 07:04:23.538645 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26099
2026-09-27T07:04:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (870 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 07:04:23.557914 21513 mini_oidc.cc:152] Starting JWKS server
I20260927 07:04:23.558956 21513 webserver.cc:460] Webserver started at https://127.0.0.1:34881/ using document root <none> and password file <none>
I20260927 07:04:23.559034 21513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:04:23.559170 21513 webserver.cc:460] Webserver started at http://127.0.0.1:38531/ using document root <none> and password file <none>
2026-09-27T07:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:23Z Disabled control of system clock
I20260927 07:04:23.566704 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45755
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45755
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/testchainca.cert with env {}
W20260927 07:04:23.644913 26625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:23.645153 26625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:23.645200 26625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:23.646355 26625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:23.646417 26625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:23.646440 26625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:23.646468 26625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:23.646520 26625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:04:23.646545 26625 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:04:23.646570 26625 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34881/jwks.json
I20260927 07:04:23.648131 26625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:23.648401 26625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:23.648597 26625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:23.653751 26633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.653751 26631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.653947 26625 server_base.cc:1034] running on GCE node
W20260927 07:04:23.653764 26630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.654274 26625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:23.654560 26625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:23.655694 26625 hybrid_clock.cc:648] HybridClock initialized: now 1790492663655683 us; error 31 us; skew 500 ppm
I20260927 07:04:23.656898 26625 webserver.cc:460] Webserver started at http://127.21.2.126:42939/ using document root <none> and password file <none>
I20260927 07:04:23.657130 26625 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:23.657174 26625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:23.657342 26625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:23.658402 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "9ecb616df8ea4365a3c07dd8613adb18"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "d647ad6327abf97757a04df7ba7c34e3"
server_key_iv: "4cf47cd479b4e1e8ef4cff94623a4578"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.658764 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "9ecb616df8ea4365a3c07dd8613adb18"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "d647ad6327abf97757a04df7ba7c34e3"
server_key_iv: "4cf47cd479b4e1e8ef4cff94623a4578"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.660152 26625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:23.660856 26639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.661022 26625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:23.661108 26625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "9ecb616df8ea4365a3c07dd8613adb18"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "d647ad6327abf97757a04df7ba7c34e3"
server_key_iv: "4cf47cd479b4e1e8ef4cff94623a4578"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.661177 26625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:23.679011 26625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:23.679454 26625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:23.679623 26625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:23.684340 26625 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:45755
I20260927 07:04:23.684365 26691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:45755 every 8 connection(s)
I20260927 07:04:23.684772 26625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:23.685683 26692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:23.688400 26692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18: Bootstrap starting.
I20260927 07:04:23.688999 26692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:23.689291 26692 log.cc:824] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:23.689993 26692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18: No bootstrap required, opened a new log
I20260927 07:04:23.690974 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26625
I20260927 07:04:23.691056 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:23.691309 21513 external_mini_cluster.cc:1428] Setting key fc6d87490d81d35d7d8a67dd90561ec9
I20260927 07:04:23.691843 26692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } }
I20260927 07:04:23.692029 26692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:23.692078 26692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecb616df8ea4365a3c07dd8613adb18, State: Initialized, Role: FOLLOWER
I20260927 07:04:23.692202 26692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } }
I20260927 07:04:23.692294 26692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:23.692337 26692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:23.692387 26692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:23.693004 26692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } }
I20260927 07:04:23.693090 26692 leader_election.cc:302] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ecb616df8ea4365a3c07dd8613adb18; no voters:
I20260927 07:04:23.693261 26692 leader_election.cc:288] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:23.693300 26697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:23.693424 26697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecb616df8ea4365a3c07dd8613adb18, State: Running, Role: LEADER
I20260927 07:04:23.693519 26692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:23.693511 26697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } }
I20260927 07:04:23.693836 26697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ecb616df8ea4365a3c07dd8613adb18. Latest consensus state: current_term: 1 leader_uuid: "9ecb616df8ea4365a3c07dd8613adb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } } }
I20260927 07:04:23.693915 26697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:23.693847 26699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ecb616df8ea4365a3c07dd8613adb18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecb616df8ea4365a3c07dd8613adb18" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45755 } } }
I20260927 07:04:23.694062 26699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:23.694926 26702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:23.695914 26702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:23.697314 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.691845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.21.2.126:45755)
0927 07:04:23.692040 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.692063 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.692095 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:04:23.692214 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.692220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.692271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.692414 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.692619 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.694154 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.695241 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.695781 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.695912 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.696629 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.696632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.696633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.696828 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.696869 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.696883 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.696942 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.697025 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.697098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260927 07:04:23.697494 26702 catalog_manager.cc:1347] Generated new cluster ID: c9648e40690e416eb110fc923acdab10
I20260927 07:04:23.697543 26702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:23.698163 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:04:23.708792 26702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:23.709306 26702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:23.718139 26702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ecb616df8ea4365a3c07dd8613adb18: Generated new TSK 0
I20260927 07:04:23.718335 26702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 07:04:23.754174 26722 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 07:04:23.754297 26722 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 07:04:23.754607 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.730496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.21.2.126:45755)
0927 07:04:23.730545 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.730547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.730663 (+ 116us) server_negotiation.cc:407] Connection header received
0927 07:04:23.730758 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.730759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.730769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.730835 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:04:23.730999 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.731547 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.732648 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.732834 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.732887 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.733126 (+ 239us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:04:23.754159 (+ 21033us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 07:04:23.754540 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 07:04:23.754674 26696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 07:04:23.754814 26721 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:23.730379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45755 (local address 127.0.0.1:51238)
0927 07:04:23.730586 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.730598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.730670 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.730855 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.730856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.730858 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:04:23.730934 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:23.730937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.731615 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.731616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:23.732562 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:23.732564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.732916 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.732917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:23.732983 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.733005 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:04:23.754331 (+ 21326us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 07:04:23.754453 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.2.126:45755: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.2.126:45755: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 07:04:23.755249 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26625
2026-09-27T07:04:23Z chronyd exiting
I20260927 07:04:23.791584 21513 test_util.cc:175] -----------------------------------------------
I20260927 07:04:23.791666 21513 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492641704908-21513-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 07:04:23.792400 21513 mini_oidc.cc:152] Starting JWKS server
I20260927 07:04:23.793056 21513 webserver.cc:460] Webserver started at https://127.0.0.1:45425/ using document root <none> and password file <none>
I20260927 07:04:23.793093 21513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:04:23.793252 21513 webserver.cc:460] Webserver started at http://127.0.0.1:42545/ using document root <none> and password file <none>
2026-09-27T07:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:23Z Disabled control of system clock
I20260927 07:04:23.801229 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36921
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:36921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45425/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/testchainca.cert with env {}
W20260927 07:04:23.880510 26728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:23.880729 26728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:23.880751 26728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:23.881954 26728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:23.881995 26728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:23.882007 26728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:23.882020 26728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:23.882032 26728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:04:23.882068 26728 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:04:23.882097 26728 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45425/jwks.json
I20260927 07:04:23.883630 26728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:36921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45425/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:23.883898 26728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:23.884107 26728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:23.888814 26733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:23.888839 26734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.889027 26728 server_base.cc:1034] running on GCE node
W20260927 07:04:23.889050 26736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:23.889283 26728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:23.889575 26728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:23.890748 26728 hybrid_clock.cc:648] HybridClock initialized: now 1790492663890738 us; error 39 us; skew 500 ppm
I20260927 07:04:23.897070 26728 webserver.cc:460] Webserver started at http://127.21.2.126:36059/ using document root <none> and password file <none>
I20260927 07:04:23.897349 26728 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:23.897424 26728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:23.897576 26728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:23.898761 26728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "145f08deadbf4b4f94bc1b90f9e3afd4"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "f623f9b8b7f14d53239d0b4aab7cf243"
server_key_iv: "ac22dac0a121d036e0dd9e14b27cff83"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.899133 26728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "145f08deadbf4b4f94bc1b90f9e3afd4"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "f623f9b8b7f14d53239d0b4aab7cf243"
server_key_iv: "ac22dac0a121d036e0dd9e14b27cff83"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.900614 26728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260927 07:04:23.901419 26742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:23.901600 26728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:23.901701 26728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "145f08deadbf4b4f94bc1b90f9e3afd4"
format_stamp: "Formatted at 2026-09-27 07:04:23 on dist-test-slave-ltqz"
server_key: "f623f9b8b7f14d53239d0b4aab7cf243"
server_key_iv: "ac22dac0a121d036e0dd9e14b27cff83"
server_key_version: "encryptionkey@0"
I20260927 07:04:23.901775 26728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:23.919632 26728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:23.919946 26728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:23.920094 26728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:23.924016 26728 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:36921
I20260927 07:04:23.924057 26794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:36921 every 8 connection(s)
I20260927 07:04:23.924410 26728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:23.925034 26795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:23.925587 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26728
I20260927 07:04:23.925711 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:23.925900 21513 external_mini_cluster.cc:1428] Setting key dc09d3929ddb677909b721608156d869
I20260927 07:04:23.928964 26795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4: Bootstrap starting.
I20260927 07:04:23.929566 26795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:23.929854 26795 log.cc:824] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:23.930649 26795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4: No bootstrap required, opened a new log
I20260927 07:04:23.932042 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.926456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.21.2.126:36921)
0927 07:04:23.926671 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.926681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.926714 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:04:23.927181 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.927189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.927270 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.927441 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:23.927662 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.929005 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.929877 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.930453 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.930615 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.931334 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:23.931338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:23.931338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:23.931562 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:23.931596 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:23.931607 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:23.931702 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:23.931792 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:04:23.931871 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260927 07:04:23.932682 26795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } }
I20260927 07:04:23.932860 26795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:23.932914 26795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145f08deadbf4b4f94bc1b90f9e3afd4, State: Initialized, Role: FOLLOWER
I20260927 07:04:23.932983 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:04:23.933043 26795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } }
I20260927 07:04:23.933112 26795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:23.933138 26795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:23.933174 26795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:23.933876 26795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } }
I20260927 07:04:23.933992 26795 leader_election.cc:302] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 145f08deadbf4b4f94bc1b90f9e3afd4; no voters:
I20260927 07:04:23.934157 26795 leader_election.cc:288] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:23.934222 26800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:23.934374 26800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [term 1 LEADER]: Becoming Leader. State: Replica: 145f08deadbf4b4f94bc1b90f9e3afd4, State: Running, Role: LEADER
I20260927 07:04:23.934405 26795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:23.934468 26800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } }
I20260927 07:04:23.934894 26801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } } }
I20260927 07:04:23.934911 26802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 145f08deadbf4b4f94bc1b90f9e3afd4. Latest consensus state: current_term: 1 leader_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145f08deadbf4b4f94bc1b90f9e3afd4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36921 } } }
I20260927 07:04:23.935009 26801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:23.935020 26802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:23.935632 26807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:23.936182 26807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:23.937659 26807 catalog_manager.cc:1347] Generated new cluster ID: d73e7cdcd27246d08fc34506acdf8161
I20260927 07:04:23.937719 26807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:23.950943 26807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:23.951651 26807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:23.965719 26807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 145f08deadbf4b4f94bc1b90f9e3afd4: Generated new TSK 0
I20260927 07:04:23.965929 26807 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 07:04:23.986716 26825 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 07:04:23.986811 26825 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 07:04:23.987145 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:23.972062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.21.2.126:36921)
0927 07:04:23.972120 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:23.972123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:23.972211 (+ 88us) server_negotiation.cc:407] Connection header received
0927 07:04:23.972288 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:23.972289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:23.972303 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:23.972372 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:04:23.972486 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.973292 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.974032 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.974357 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.974431 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.974625 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:04:23.986911 (+ 12286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 07:04:23.987069 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 07:04:23.987207 26798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 07:04:23.987216 26824 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:23.971986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:36921 (local address 127.0.0.1:40738)
0927 07:04:23.972152 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:23.972160 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:23.972210 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:23.972381 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:23.972382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:23.972384 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:04:23.972430 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:23.972432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.973382 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.973382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:23.973955 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:23.973957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:23.974455 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:23.974456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:23.974514 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:23.974519 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:04:23.987022 (+ 12503us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 07:04:23.987124 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.2.126:36921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45425/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260927 07:04:23.987511 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26728
2026-09-27T07:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 07:04:24.024914 21513 mini_oidc.cc:152] Starting JWKS server
I20260927 07:04:24.025599 21513 webserver.cc:460] Webserver started at https://127.0.0.1:42329/ using document root <none> and password file <none>
I20260927 07:04:24.025665 21513 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:04:24.025861 21513 webserver.cc:460] Webserver started at http://127.0.0.1:34295/ using document root <none> and password file <none>
2026-09-27T07:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:24Z Disabled control of system clock
I20260927 07:04:24.033071 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:37653
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:37653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42329/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 07:04:24.112533 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:24.112787 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:24.112823 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:24.113971 26831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:24.114039 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:24.114061 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:24.114082 26831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:24.114108 26831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:04:24.114130 26831 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:04:24.114159 26831 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42329/jwks.json
I20260927 07:04:24.115558 26831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:37653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42329/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:24.115839 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:24.116117 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:24.120648 26839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:24.120675 26837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:24.120692 26836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:24.120971 26831 server_base.cc:1034] running on GCE node
I20260927 07:04:24.121351 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:24.121613 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:24.122764 26831 hybrid_clock.cc:648] HybridClock initialized: now 1790492664122753 us; error 39 us; skew 500 ppm
I20260927 07:04:24.124040 26831 webserver.cc:460] Webserver started at http://127.21.2.126:42849/ using document root <none> and password file <none>
I20260927 07:04:24.124321 26831 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:24.124393 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:24.124533 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:24.125744 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "affa06db557843448c668ed5c72cd8ac"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "2e7054d507c3e0330f344bdf5931ca71"
server_key_iv: "52484854c50c2559a181e56ebdfc47cb"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.126108 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "affa06db557843448c668ed5c72cd8ac"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "2e7054d507c3e0330f344bdf5931ca71"
server_key_iv: "52484854c50c2559a181e56ebdfc47cb"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.127575 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:24.128335 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:24.128531 26831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:24.128629 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "affa06db557843448c668ed5c72cd8ac"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "2e7054d507c3e0330f344bdf5931ca71"
server_key_iv: "52484854c50c2559a181e56ebdfc47cb"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.128702 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:24.142661 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:24.142995 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:24.143142 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:24.146898 26831 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:37653
I20260927 07:04:24.146965 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:37653 every 8 connection(s)
I20260927 07:04:24.147306 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:24.148003 26898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:24.150738 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac: Bootstrap starting.
I20260927 07:04:24.151554 26898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:24.151850 26898 log.cc:824] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:24.152659 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac: No bootstrap required, opened a new log
I20260927 07:04:24.154278 26898 raft_consensus.cc:348] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } }
I20260927 07:04:24.154448 26898 raft_consensus.cc:374] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:24.154495 26898 raft_consensus.cc:729] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa06db557843448c668ed5c72cd8ac, State: Initialized, Role: FOLLOWER
I20260927 07:04:24.154641 26898 consensus_queue.cc:260] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } }
I20260927 07:04:24.154718 26898 raft_consensus.cc:388] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:24.154762 26898 raft_consensus.cc:482] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:24.154810 26898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:24.155423 26898 raft_consensus.cc:504] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } }
I20260927 07:04:24.155540 26898 leader_election.cc:302] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affa06db557843448c668ed5c72cd8ac; no voters:
I20260927 07:04:24.155716 26898 leader_election.cc:288] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:24.155766 26901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:24.155931 26901 raft_consensus.cc:686] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [term 1 LEADER]: Becoming Leader. State: Replica: affa06db557843448c668ed5c72cd8ac, State: Running, Role: LEADER
I20260927 07:04:24.155977 26898 sys_catalog.cc:564] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:24.156051 26901 consensus_queue.cc:237] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } }
I20260927 07:04:24.156414 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader affa06db557843448c668ed5c72cd8ac. Latest consensus state: current_term: 1 leader_uuid: "affa06db557843448c668ed5c72cd8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } } }
I20260927 07:04:24.156503 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:24.156610 26902 sys_catalog.cc:455] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "affa06db557843448c668ed5c72cd8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affa06db557843448c668ed5c72cd8ac" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 37653 } } }
I20260927 07:04:24.156675 26902 sys_catalog.cc:458] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:24.157102 26909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:24.157352 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26831
I20260927 07:04:24.157462 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:24.157613 26909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:24.157640 21513 external_mini_cluster.cc:1428] Setting key 045a7eff2de9ca19251e61f5731be05b
I20260927 07:04:24.159312 26909 catalog_manager.cc:1347] Generated new cluster ID: 87f60f1c026f45f4b1091cea8a09b345
I20260927 07:04:24.159389 26909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:24.163910 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:24.158050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.21.2.126:37653)
0927 07:04:24.158709 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:24.158715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:24.158757 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:04:24.158899 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:24.158906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:24.158972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:24.159106 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:24.159395 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.160710 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.161829 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.162411 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.162517 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.163220 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:24.163224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:24.163224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:24.163450 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:24.163490 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:24.163503 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:24.163564 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:24.163653 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 07:04:24.163722 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":82,"threads_started":1}
I20260927 07:04:24.165040 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:04:24.173067 26909 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:24.173555 26909 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:24.175457 26909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P affa06db557843448c668ed5c72cd8ac: Generated new TSK 0
I20260927 07:04:24.175621 26909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:04:24.185201 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:24.181681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.21.2.126:37653)
0927 07:04:24.181720 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:24.181722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:24.181813 (+ 91us) server_negotiation.cc:407] Connection header received
0927 07:04:24.181900 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:24.181901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:24.181911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:24.181959 (+ 48us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:04:24.182106 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.182591 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.183711 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.183901 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.183953 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.184151 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:04:24.185011 (+ 860us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42329/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 07:04:24.185136 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42329/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 07:04:24.185256 26920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42329/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 07:04:24.185621 26927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:24.181582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:37653 (local address 127.0.0.1:58992)
0927 07:04:24.181734 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:24.181748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:24.181833 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:24.181975 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:24.181976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:24.181978 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:04:24.182044 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:24.182046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.182678 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.182679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:24.183638 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:24.183640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.183976 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.183977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:24.184061 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.184067 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:04:24.185102 (+ 1035us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42329/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 07:04:24.185199 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.2.126:37653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42329/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260927 07:04:24.185775 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26831
E20260927 07:04:24.195994 26928 webserver.cc:533] Webserver: sslize failed:
E20260927 07:04:24.196084 26928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T07:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (198 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T07:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:24Z Disabled control of system clock
I20260927 07:04:24.230185 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34865
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34865
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/testchainca.cert with env {}
W20260927 07:04:24.308514 26932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:24.308712 26932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:24.308734 26932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:24.309906 26932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:24.309947 26932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:24.309959 26932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:24.309973 26932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:24.310027 26932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:04:24.310050 26932 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:04:24.310076 26932 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 07:04:24.311580 26932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:24.311836 26932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:24.312048 26932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:24.316886 26938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:24.317044 26932 server_base.cc:1034] running on GCE node
W20260927 07:04:24.316867 26940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:24.316862 26937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:24.317407 26932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:24.317696 26932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:24.318856 26932 hybrid_clock.cc:648] HybridClock initialized: now 1790492664318841 us; error 35 us; skew 500 ppm
I20260927 07:04:24.320062 26932 webserver.cc:460] Webserver started at http://127.21.2.126:34713/ using document root <none> and password file <none>
I20260927 07:04:24.320308 26932 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:24.320375 26932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:24.320511 26932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:24.321524 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "a2e4df1494a9470da38bee5f8da2d045"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "e0db157a205a0bbf239ba92e8da99f2d"
server_key_iv: "07ab16ba82c4dcd7ee83c76497b446d8"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.321894 26932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "a2e4df1494a9470da38bee5f8da2d045"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "e0db157a205a0bbf239ba92e8da99f2d"
server_key_iv: "07ab16ba82c4dcd7ee83c76497b446d8"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.323288 26932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:24.324296 26946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:24.324465 26932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:24.324556 26932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "a2e4df1494a9470da38bee5f8da2d045"
format_stamp: "Formatted at 2026-09-27 07:04:24 on dist-test-slave-ltqz"
server_key: "e0db157a205a0bbf239ba92e8da99f2d"
server_key_iv: "07ab16ba82c4dcd7ee83c76497b446d8"
server_key_version: "encryptionkey@0"
I20260927 07:04:24.324637 26932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:24.338428 26932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:24.338745 26932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:24.338897 26932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:24.342847 26932 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34865
I20260927 07:04:24.342918 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34865 every 8 connection(s)
I20260927 07:04:24.343488 26932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:24.344435 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 26932
I20260927 07:04:24.344457 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:24.344564 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:24.344741 21513 external_mini_cluster.cc:1428] Setting key caf13f500a70219509b18304a783b507
I20260927 07:04:24.347913 26999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: Bootstrap starting.
I20260927 07:04:24.348524 26999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:24.348825 26999 log.cc:824] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:24.349579 26999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: No bootstrap required, opened a new log
I20260927 07:04:24.350966 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:24.345250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.21.2.126:34865)
0927 07:04:24.345500 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:24.345509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:24.345541 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:04:24.346043 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:24.346052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:24.346136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:24.346293 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:24.346504 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.347882 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.348747 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.349291 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.349425 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.350159 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:24.350164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:24.350164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:04:24.350367 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:24.350409 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:04:24.350423 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:24.350495 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:24.350585 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:04:24.350662 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
I20260927 07:04:24.352172 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:04:24.352473 26999 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } }
I20260927 07:04:24.352658 26999 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:24.352694 26999 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e4df1494a9470da38bee5f8da2d045, State: Initialized, Role: FOLLOWER
I20260927 07:04:24.352820 26999 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } }
I20260927 07:04:24.352898 26999 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:24.352931 26999 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:24.352977 26999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:24.353569 26999 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } }
I20260927 07:04:24.353658 26999 leader_election.cc:302] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e4df1494a9470da38bee5f8da2d045; no voters:
I20260927 07:04:24.353833 26999 leader_election.cc:288] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:24.353948 27004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:24.354084 26999 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:24.354120 27004 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [term 1 LEADER]: Becoming Leader. State: Replica: a2e4df1494a9470da38bee5f8da2d045, State: Running, Role: LEADER
I20260927 07:04:24.354226 27004 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } }
I20260927 07:04:24.354625 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e4df1494a9470da38bee5f8da2d045. Latest consensus state: current_term: 1 leader_uuid: "a2e4df1494a9470da38bee5f8da2d045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } } }
I20260927 07:04:24.354712 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:24.354638 27006 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e4df1494a9470da38bee5f8da2d045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e4df1494a9470da38bee5f8da2d045" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34865 } } }
I20260927 07:04:24.354779 27006 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045 [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:24.355957 27020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:24.356002 27020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:24.356065 27013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:24.356209 27013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:24.357545 27013 catalog_manager.cc:1347] Generated new cluster ID: 9fa60e063b0a4dbf87a72af7f7692d09
I20260927 07:04:24.357599 27013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:24.371284 27013 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:24.371835 27013 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:24.375821 27013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e4df1494a9470da38bee5f8da2d045: Generated new TSK 0
I20260927 07:04:24.375996 27013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:04:24.815510 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:24.387006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.21.2.126:34865)
0927 07:04:24.387081 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:24.387083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:24.387193 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:04:24.387335 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:24.387336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:24.387350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:24.387417 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:04:24.387645 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.388470 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.389636 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.389840 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.389893 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.390077 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:04:24.815222 (+425145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 07:04:24.815444 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36300: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 07:04:24.815577 27002 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36300: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 07:04:24.815711 27028 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:24.386951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34865 (local address 127.0.0.1:36300)
0927 07:04:24.387124 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:24.387136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:24.387242 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:24.387487 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:24.387488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:24.387490 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:04:24.387550 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:24.387553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.388576 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.388577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:24.389538 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:24.389541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:24.389915 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:24.389917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:24.389979 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:24.389985 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:04:24.815434 (+425449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 07:04:24.815561 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.2.126:34865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260927 07:04:24.816054 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 26932
2026-09-27T07:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (610 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:25 dist-test-slave-ltqz krb5kdc[27040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:25 dist-test-slave-ltqz krb5kdc[27040](info): set up 2 sockets
Sep 27 07:04:25 dist-test-slave-ltqz krb5kdc[27040](info): commencing operation
krb5kdc: starting...
W20260927 07:04:26.031580 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:04:26.149919 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34391
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:26.231025 27053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:26.231261 27053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:26.231297 27053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:26.232475 27053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:26.232550 27053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:26.232574 27053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:26.232595 27053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:26.232623 27053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:26.234059 27053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:26.234330 27053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:26.234550 27053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:26.239228 27059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.239248 27061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.239257 27058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:26.239403 27053 server_base.cc:1034] running on GCE node
I20260927 07:04:26.239719 27053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:26.240010 27053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:26.241178 27053 hybrid_clock.cc:648] HybridClock initialized: now 1790492666241160 us; error 30 us; skew 500 ppm
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.243625 27053 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:04:26.244001 27053 webserver.cc:460] Webserver started at http://127.21.2.126:35049/ using document root <none> and password file <none>
I20260927 07:04:26.244220 27053 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:26.244294 27053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:26.244388 27053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:26.245373 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "f3f9216f84ea4968a7a51b39ca04dd7c"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "a619c56993cb9cf55d935e884e5dabcf"
server_key_iv: "c13f7a76efa0328613d4f3bc5efc5fef"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.245664 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "f3f9216f84ea4968a7a51b39ca04dd7c"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "a619c56993cb9cf55d935e884e5dabcf"
server_key_iv: "c13f7a76efa0328613d4f3bc5efc5fef"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.246913 27053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:04:26.247699 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.247906 27053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:26.247988 27053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "f3f9216f84ea4968a7a51b39ca04dd7c"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "a619c56993cb9cf55d935e884e5dabcf"
server_key_iv: "c13f7a76efa0328613d4f3bc5efc5fef"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.248042 27053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:26.264616 27053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:26.266855 27053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:26.266978 27053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:26.271010 27053 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34391
I20260927 07:04:26.271060 27120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34391 every 8 connection(s)
I20260927 07:04:26.271499 27053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:26.272405 27121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:26.274559 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c: Bootstrap starting.
I20260927 07:04:26.274961 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27053
I20260927 07:04:26.275070 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:26.275245 27121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:26.275266 21513 external_mini_cluster.cc:1428] Setting key 8c33ef43b9e1b6df77b974a2647781e5
I20260927 07:04:26.275611 27121 log.cc:824] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:26.276391 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c: No bootstrap required, opened a new log
I20260927 07:04:26.278788 27121 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } }
I20260927 07:04:26.278970 27121 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:26.279021 27121 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f9216f84ea4968a7a51b39ca04dd7c, State: Initialized, Role: FOLLOWER
I20260927 07:04:26.279150 27121 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } }
I20260927 07:04:26.279241 27121 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:26.279284 27121 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:26.279333 27121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:26.279989 27121 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } }
I20260927 07:04:26.280093 27121 leader_election.cc:302] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f9216f84ea4968a7a51b39ca04dd7c; no voters:
I20260927 07:04:26.280252 27121 leader_election.cc:288] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:26.280315 27126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:26.280504 27121 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:26.280506 27126 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [term 1 LEADER]: Becoming Leader. State: Replica: f3f9216f84ea4968a7a51b39ca04dd7c, State: Running, Role: LEADER
I20260927 07:04:26.280730 27126 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } }
I20260927 07:04:26.281162 27127 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } } }
I20260927 07:04:26.281291 27127 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:26.281657 27128 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f9216f84ea4968a7a51b39ca04dd7c. Latest consensus state: current_term: 1 leader_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f9216f84ea4968a7a51b39ca04dd7c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34391 } } }
I20260927 07:04:26.281734 27128 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:26.281941 27135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:26.282387 27135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:26.286972 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.275782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.21.2.126:34391)
0927 07:04:26.275932 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.275941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.275971 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:04:26.276595 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.276603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.276662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.276813 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.277379 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.278810 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.279698 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.280338 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.280766 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.283505 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.283509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.283509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.283521 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.285394 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.285750 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.285753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.285753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.285863 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.286052 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.286053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.286053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.286380 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.286539 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.286606 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.286743 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 07:04:26.287351 27135 catalog_manager.cc:1347] Generated new cluster ID: 394bfcf3c65a4c4cbf2ab3c8fd708e03
I20260927 07:04:26.287460 27135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:26.299679 27135 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:26.300207 27135 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:26.307960 27135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f9216f84ea4968a7a51b39ca04dd7c: Generated new TSK 0
I20260927 07:04:26.308138 27135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:04:26.338501 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:26.421221 27147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:26.421489 27147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:26.421532 27147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:26.422749 27147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:26.422818 27147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:26.422864 27147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:26.424482 27147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:26.424754 27147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:26.424989 27147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:26.429903 27153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.429903 27152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.429881 27155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:26.430047 27147 server_base.cc:1034] running on GCE node
I20260927 07:04:26.430315 27147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:26.430563 27147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:26.431696 27147 hybrid_clock.cc:648] HybridClock initialized: now 1790492666431677 us; error 37 us; skew 500 ppm
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.434058 27147 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:04:26.434518 27147 webserver.cc:460] Webserver started at http://127.21.2.65:41295/ using document root <none> and password file <none>
I20260927 07:04:26.434731 27147 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:26.434775 27147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:26.434926 27147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:26.436008 27147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "a2d9d7ab3b914b9388cbdb50cc0c36ac"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "f64fc1f458eda5af0694ed16ce1eac6a"
server_key_iv: "73919017c4481ef76c44c970548bb0b6"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.436350 27147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d9d7ab3b914b9388cbdb50cc0c36ac"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "f64fc1f458eda5af0694ed16ce1eac6a"
server_key_iv: "73919017c4481ef76c44c970548bb0b6"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.437651 27147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:26.438416 27162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.438580 27147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:26.438679 27147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "a2d9d7ab3b914b9388cbdb50cc0c36ac"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "f64fc1f458eda5af0694ed16ce1eac6a"
server_key_iv: "73919017c4481ef76c44c970548bb0b6"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.438748 27147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:26.451177 27147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:26.453154 27147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:26.453318 27147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:26.453572 27147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:26.453888 27147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:26.453939 27147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.453975 27147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:26.454017 27147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.459993 27147 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:44595
I20260927 07:04:26.460040 27275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:44595 every 8 connection(s)
I20260927 07:04:26.460353 27147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:26.463617 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27147
I20260927 07:04:26.463730 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:26.463908 21513 external_mini_cluster.cc:1428] Setting key dc65ebde72c78f852cbec73ce4348640
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:26.470155 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.461076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:40211 (local address 127.21.2.126:34391)
0927 07:04:26.461142 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.461144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.461327 (+ 183us) server_negotiation.cc:407] Connection header received
0927 07:04:26.462256 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.462257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.462267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.462335 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.462941 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.463481 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.464765 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.464992 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.465064 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.467408 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.467410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.467411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.467418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.468801 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.469205 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.469206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.469207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.469267 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.469520 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.469521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.469522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.469639 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.469707 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.469977 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.470063 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:04:26.470369 27278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.461022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34391 (local address 127.21.2.65:40211)
0927 07:04:26.461201 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:26.461242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:26.462136 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:26.462414 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:26.462417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:26.462661 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:26.462849 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.462852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.463591 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.463593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.464667 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.464670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.465071 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.465072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.465207 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.465779 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:26.465788 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:26.467283 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:26.468899 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.468906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.468908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.469129 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.469331 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.469332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.469333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.469448 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.469709 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:26.469716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:26.469851 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 07:04:26.469953 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 07:04:26.470108 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260927 07:04:26.470935 27276 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34391
I20260927 07:04:26.471055 27276 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:26.471263 27276 heartbeater.cc:511] Master 127.21.2.126:34391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:04:26.471947 27085 ts_manager.cc:194] Registered new tserver with Master: a2d9d7ab3b914b9388cbdb50cc0c36ac (127.21.2.65:44595)
I20260927 07:04:26.472584 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:40211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:04:26.490685 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:26.573277 27281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:26.573565 27281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:26.573607 27281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:26.574870 27281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:26.574954 27281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:26.575012 27281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:26.576715 27281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:26.577008 27281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:26.577250 27281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:26.582283 27289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.582345 27287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.582299 27286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:26.582623 27281 server_base.cc:1034] running on GCE node
I20260927 07:04:26.582841 27281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:26.583106 27281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:26.584266 27281 hybrid_clock.cc:648] HybridClock initialized: now 1790492666584260 us; error 36 us; skew 500 ppm
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.586786 27281 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:04:26.587268 27281 webserver.cc:460] Webserver started at http://127.21.2.66:34483/ using document root <none> and password file <none>
I20260927 07:04:26.587577 27281 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:26.587651 27281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:26.587790 27281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:26.589011 27281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "4acfc859bde145dba23178e56a2cbeb0"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "0f626bdf45a00b746f2659710eb14527"
server_key_iv: "e6ec4e21240047efc9c40270531ef107"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.589370 27281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "4acfc859bde145dba23178e56a2cbeb0"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "0f626bdf45a00b746f2659710eb14527"
server_key_iv: "e6ec4e21240047efc9c40270531ef107"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.590714 27281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:26.591487 27296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.591668 27281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:26.591763 27281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "4acfc859bde145dba23178e56a2cbeb0"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "0f626bdf45a00b746f2659710eb14527"
server_key_iv: "e6ec4e21240047efc9c40270531ef107"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.591845 27281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:26.601573 27281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:26.603840 27281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:26.604000 27281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:26.604224 27281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:26.604537 27281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:26.604588 27281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.604628 27281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:26.604658 27281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.610899 27281 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:36933
I20260927 07:04:26.610955 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:36933 every 8 connection(s)
I20260927 07:04:26.611267 27281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:26.615875 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27281
I20260927 07:04:26.615988 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:26.616209 21513 external_mini_cluster.cc:1428] Setting key 254841f56f8a215e450c735b249b6f0d
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:26.621624 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.612040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:58925 (local address 127.21.2.126:34391)
0927 07:04:26.612111 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.612113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.612337 (+ 224us) server_negotiation.cc:407] Connection header received
0927 07:04:26.613253 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.613255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.613265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.613329 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.613993 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.614500 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.615570 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.615748 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.615802 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.618599 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.618601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.618601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.618611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.620243 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.620643 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.620644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.620645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.620711 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.620952 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.620953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.620954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.621098 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.621167 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.621409 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.621501 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:04:26.621649 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.611962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34391 (local address 127.21.2.66:58925)
0927 07:04:26.612180 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:26.612222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:26.613121 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:26.613411 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:26.613415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:26.613687 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:26.613900 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.613904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.614582 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.614583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.615491 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.615494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.615838 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.615839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.615997 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.616642 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:26.616651 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:26.618499 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:26.620331 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.620336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.620338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.620569 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.620782 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.620783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.620783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.620886 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.621168 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:26.621172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:26.621285 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 07:04:26.621362 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:04:26.621485 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260927 07:04:26.622164 27410 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34391
I20260927 07:04:26.622272 27410 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:26.622448 27410 heartbeater.cc:511] Master 127.21.2.126:34391 requested a full tablet report, sending...
I20260927 07:04:26.622856 27085 ts_manager.cc:194] Registered new tserver with Master: 4acfc859bde145dba23178e56a2cbeb0 (127.21.2.66:36933)
I20260927 07:04:26.623437 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:58925
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:04:26.640625 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:26.720253 27415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:26.720501 27415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:26.720538 27415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:26.721681 27415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:26.721745 27415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:26.721789 27415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:26.723248 27415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:26.723537 27415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:26.723754 27415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:04:26.728694 27415 server_base.cc:1034] running on GCE node
W20260927 07:04:26.728773 27420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.728807 27423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:26.728715 27421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:26.729079 27415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:26.729303 27415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:26.730449 27415 hybrid_clock.cc:648] HybridClock initialized: now 1790492666730431 us; error 37 us; skew 500 ppm
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.732908 27415 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:04:26.733335 27415 webserver.cc:460] Webserver started at http://127.21.2.67:35463/ using document root <none> and password file <none>
I20260927 07:04:26.733520 27415 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:26.733559 27415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:26.733646 27415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:26.734649 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "b0c7208185c5402e8021e6aac5ba9a98"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "3e573bd793b275cbcb85810f3a7bec19"
server_key_iv: "ba8a0cf5eb520718cf4b183a6049f55a"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.734949 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c7208185c5402e8021e6aac5ba9a98"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "3e573bd793b275cbcb85810f3a7bec19"
server_key_iv: "ba8a0cf5eb520718cf4b183a6049f55a"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.736351 27415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:26.737068 27430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.737231 27415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:26.737337 27415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "b0c7208185c5402e8021e6aac5ba9a98"
format_stamp: "Formatted at 2026-09-27 07:04:26 on dist-test-slave-ltqz"
server_key: "3e573bd793b275cbcb85810f3a7bec19"
server_key_iv: "ba8a0cf5eb520718cf4b183a6049f55a"
server_key_version: "encryptionkey@0"
I20260927 07:04:26.737409 27415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:26.767339 27415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:26.769685 27415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:26.769847 27415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:26.770083 27415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:26.770406 27415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:26.770462 27415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.770504 27415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:26.770530 27415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:26.776536 27415 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:45719
I20260927 07:04:26.776582 27543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:45719 every 8 connection(s)
I20260927 07:04:26.776906 27415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:04:26.785827 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27415
I20260927 07:04:26.785938 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:26.786170 21513 external_mini_cluster.cc:1428] Setting key 147d11fdb9985fe1e1afab251051c633
I20260927 07:04:26.786404 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.777633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:47699 (local address 127.21.2.126:34391)
0927 07:04:26.777680 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.777682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.777890 (+ 208us) server_negotiation.cc:407] Connection header received
0927 07:04:26.778792 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.778793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.778803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.778840 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.779533 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.780031 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.781054 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.781248 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.781366 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.783755 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.783756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.783756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.783763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.784960 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.785359 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.785360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.785360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.785410 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.785664 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.785665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.785665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.785766 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.785815 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.786193 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.786286 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:04:26.786657 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.777612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34391 (local address 127.21.2.67:47699)
0927 07:04:26.777768 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:26.777803 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:26.778690 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:26.778900 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:26.778903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:26.779155 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:26.779351 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.779355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.780105 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.780106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.780978 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:26.780980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.781373 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.781374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:26.781497 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.782144 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:26.782153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:26.783655 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:26.785036 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.785041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.785043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.785287 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.785475 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:26.785475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:26.785475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:26.785595 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:26.785831 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:26.785838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:26.786023 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 07:04:26.786170 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 07:04:26.786385 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260927 07:04:26.787119 27544 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34391
I20260927 07:04:26.787245 27544 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:26.787472 27544 heartbeater.cc:511] Master 127.21.2.126:34391 requested a full tablet report, sending...
I20260927 07:04:26.787856 27085 ts_manager.cc:194] Registered new tserver with Master: b0c7208185c5402e8021e6aac5ba9a98 (127.21.2.67:45719)
I20260927 07:04:26.788254 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:47699
I20260927 07:04:26.798020 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/test-admin.keytab.
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.813796 21513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 07:04:26.815613 27551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:26.814727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.21.2.65:44595)
0927 07:04:26.814904 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.814909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.814917 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:04:26.814942 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.814944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.814957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.815043 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.815521 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52908: BlockingRecv error: recv got EOF from 127.0.0.1:52908 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
W20260927 07:04:26.815764 21513 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc' of type FILE
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.816807 21513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.817672 21513 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:26.825596 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.818202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.21.2.65:44595)
0927 07:04:26.818259 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.818261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.818328 (+ 67us) server_negotiation.cc:407] Connection header received
0927 07:04:26.818414 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.818415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.818431 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.818487 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.818884 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.819674 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.820387 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.820709 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.820787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.822864 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.822878 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.822878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.822885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.824419 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.824777 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.824784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.824785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.824844 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.825035 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.825036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.825036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.825176 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.825351 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.825400 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.825484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 07:04:26.827780 27551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:26.826866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.21.2.65:44595)
0927 07:04:26.826949 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.826951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.827040 (+ 89us) server_negotiation.cc:407] Connection header received
0927 07:04:26.827144 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.827145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.827157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.827228 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.827717 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52926: BlockingRecv error: recv got EOF from 127.0.0.1:52926 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 07:04:26.827867 21513 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc' of type FILE
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.828791 21513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.829602 21513 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:26.836534 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.830067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.21.2.65:44595)
0927 07:04:26.830120 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.830122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.830186 (+ 64us) server_negotiation.cc:407] Connection header received
0927 07:04:26.830275 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.830276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.830285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.830347 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.830699 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.831187 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.831921 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.832122 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.832193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.834041 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.834042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.834042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.834050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.835435 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.835778 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.835780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.835780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.835836 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.836041 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.836042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.836043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.836138 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.836232 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.836360 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.836432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 07:04:26.837899 27551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:04:26.837449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.21.2.65:44595)
0927 07:04:26.837486 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.837487 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.837529 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:04:26.837576 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.837576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.837583 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.837625 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.837854 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52942: BlockingRecv error: recv got EOF from 127.0.0.1:52942 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:04:26.837986 21513 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492641704908-21513-0/krb5kdc/krb5cc' of type FILE
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.838925 21513 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:26.839818 21513 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.65@KRBTEST.COM
I20260927 07:04:26.846657 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:26.840260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.21.2.65:44595)
0927 07:04:26.840329 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:26.840331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:26.840340 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:26.840363 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:26.840364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:26.840371 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:26.840423 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:26.840763 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.841278 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.841938 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:26.842134 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:26.842203 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:26.844210 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:26.844212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:26.844212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:26.844218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:26.845586 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.845985 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.845986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.845986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.846031 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:26.846226 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:26.846226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:26.846226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:26.846308 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:26.846371 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:26.846547 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 07:04:26.846600 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 07:04:26.847291 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27147
I20260927 07:04:26.853358 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27281
I20260927 07:04:26.858098 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27415
I20260927 07:04:26.862321 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27053
2026-09-27T07:04:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1871 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27564](info): set up 2 sockets
Sep 27 07:04:26 dist-test-slave-ltqz krb5kdc[27564](info): commencing operation
krb5kdc: starting...
W20260927 07:04:27.891516 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:04:27 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:04:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab.
Entry for principal oryx/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab.
I20260927 07:04:28.009438 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45915
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab
--principal=oryx/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:28.088495 27577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:28.088742 27577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:28.088778 27577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:28.089916 27577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:28.089989 27577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:28.090014 27577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:28.090035 27577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:28.090063 27577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:28.091573 27577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab
--principal=oryx/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:28.091853 27577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:28.092077 27577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.096683 27582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.096735 27583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.096879 27577 server_base.cc:1034] running on GCE node
W20260927 07:04:28.096662 27585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.097160 27577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.097455 27577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.098615 27577 hybrid_clock.cc:648] HybridClock initialized: now 1790492668098607 us; error 34 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.100972 27577 init.cc:377] Logged in from keytab as oryx/127.21.2.126@KRBTEST.COM (short username oryx)
I20260927 07:04:28.101363 27577 webserver.cc:460] Webserver started at http://127.21.2.126:35983/ using document root <none> and password file <none>
I20260927 07:04:28.101590 27577 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.101655 27577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.101794 27577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:28.102793 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "287e9eeb51094b37844e0a0ad282e4ec"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "3e50dcd8b12563eed985a218074b96a0"
server_key_iv: "56974c05ceab548f9c37b42b5c934388"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.103137 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "287e9eeb51094b37844e0a0ad282e4ec"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "3e50dcd8b12563eed985a218074b96a0"
server_key_iv: "56974c05ceab548f9c37b42b5c934388"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.104475 27577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:28.105157 27592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.105305 27577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:28.105399 27577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "287e9eeb51094b37844e0a0ad282e4ec"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "3e50dcd8b12563eed985a218074b96a0"
server_key_iv: "56974c05ceab548f9c37b42b5c934388"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.105479 27577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.116266 27577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.118135 27577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:28.118287 27577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.122246 27577 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:45915
I20260927 07:04:28.122278 27644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:45915 every 8 connection(s)
I20260927 07:04:28.122635 27577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:04:28.123675 27645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:28.123750 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27577
I20260927 07:04:28.123878 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:04:28.124073 21513 external_mini_cluster.cc:1428] Setting key 147af6f29b0f49c4f3af88322d61bc8a
I20260927 07:04:28.126984 27645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Bootstrap starting.
I20260927 07:04:28.127758 27645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:28.128047 27645 log.cc:824] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:28.128765 27645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: No bootstrap required, opened a new log
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:04:28.131227 27645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } }
I20260927 07:04:28.131443 27645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:28.131482 27645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e9eeb51094b37844e0a0ad282e4ec, State: Initialized, Role: FOLLOWER
I20260927 07:04:28.131613 27645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } }
I20260927 07:04:28.131693 27645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:04:28.131716 27645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:04:28.131745 27645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:28.132285 27645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } }
I20260927 07:04:28.132403 27645 leader_election.cc:302] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287e9eeb51094b37844e0a0ad282e4ec; no voters:
I20260927 07:04:28.132577 27645 leader_election.cc:288] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:04:28.132617 27650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:28.132758 27650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 LEADER]: Becoming Leader. State: Replica: 287e9eeb51094b37844e0a0ad282e4ec, State: Running, Role: LEADER
I20260927 07:04:28.132838 27645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:28.132840 27650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } }
I20260927 07:04:28.133183 27650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287e9eeb51094b37844e0a0ad282e4ec. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } }
I20260927 07:04:28.133272 27650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:28.133196 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } }
I20260927 07:04:28.133445 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:28.134580 27665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:28.134684 27665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:28.134778 27656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:04:28.134941 27656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:04:28.135154 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.124573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.21.2.126:45915)
0927 07:04:28.124744 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.124752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.124775 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:04:28.125274 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.125281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.125328 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.125437 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.126243 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.127111 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.128204 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.128606 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.128698 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.131200 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.131204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.131204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.131217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.133357 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.133740 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.133742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.133742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.133815 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.134035 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.134036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.134036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.134346 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.134493 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.134792 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.134929 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 07:04:28.136502 27656 catalog_manager.cc:1347] Generated new cluster ID: bedb1b25816649b2ba279fd49d78ea2f
I20260927 07:04:28.136554 27656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:04:28.162875 27656 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:04:28.163481 27656 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:04:28.168198 27656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Generated new TSK 0
I20260927 07:04:28.168382 27656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
I20260927 07:04:28.200116 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:28.280390 27671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:28.280596 27671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:28.280618 27671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:28.281702 27671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:28.281736 27671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:28.281762 27671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:28.283140 27671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:28.283321 27671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:28.283569 27671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.288285 27679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.288295 27676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.288542 27677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.288556 27671 server_base.cc:1034] running on GCE node
I20260927 07:04:28.288763 27671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.289006 27671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.290155 27671 hybrid_clock.cc:648] HybridClock initialized: now 1790492668290137 us; error 31 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.292551 27671 init.cc:377] Logged in from keytab as oryx/127.21.2.65@KRBTEST.COM (short username oryx)
I20260927 07:04:28.293011 27671 webserver.cc:460] Webserver started at http://127.21.2.65:45979/ using document root <none> and password file <none>
I20260927 07:04:28.293279 27671 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.293346 27671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.293480 27671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:28.294601 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "435025c5658f4879931d3d90a806516e"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "200386eb6cb9473fc5a945c4e0ae91a1"
server_key_iv: "a30f120c22ed3b435960e4cc260f6404"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.294948 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "435025c5658f4879931d3d90a806516e"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "200386eb6cb9473fc5a945c4e0ae91a1"
server_key_iv: "a30f120c22ed3b435960e4cc260f6404"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.296324 27671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:04:28.297037 27686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.297194 27671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.297272 27671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "435025c5658f4879931d3d90a806516e"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "200386eb6cb9473fc5a945c4e0ae91a1"
server_key_iv: "a30f120c22ed3b435960e4cc260f6404"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.297338 27671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.304993 27671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.306978 27671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:28.307085 27671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.307251 27671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:28.307590 27671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:28.307623 27671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.307644 27671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:28.307657 27671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.313339 27671 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:34707
I20260927 07:04:28.313486 27799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:34707 every 8 connection(s)
I20260927 07:04:28.313740 27671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:28.314607 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27671
I20260927 07:04:28.314718 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:04:28.314903 21513 external_mini_cluster.cc:1428] Setting key 0a29acc146936d15ef836feeca84bb8b
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:04:28.324295 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.314404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47027 (local address 127.21.2.126:45915)
0927 07:04:28.314469 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.314471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.314658 (+ 187us) server_negotiation.cc:407] Connection header received
0927 07:04:28.315276 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.315277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.315289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.315390 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.316344 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.317054 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.318574 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.318768 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.318825 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.321301 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.321303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.321303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.321312 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.322973 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.323357 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.323358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.323359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.323463 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.323726 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.323727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.323728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.323877 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.323937 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.324165 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.324230 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:04:28.324488 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.314383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:47027)
0927 07:04:28.314540 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.314579 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.315168 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.315457 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.315465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.316066 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.316248 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.316252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.317144 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.317146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.318495 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.318499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.318847 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.318848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.318999 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.319600 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.319612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.321207 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.323065 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.323070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.323072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.323284 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.323541 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.323542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.323542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.323659 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.323942 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.323946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.324058 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.324139 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.324255 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 07:04:28.324955 27800 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:28.325040 27800 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:28.325179 27800 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:28.325627 27609 ts_manager.cc:194] Registered new tserver with Master: 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.66@KRBTEST.COM" created.
I20260927 07:04:28.326411 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:47027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
I20260927 07:04:28.340226 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:28.420147 27805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:28.420483 27805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:28.420526 27805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:28.422740 27805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:28.422791 27805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:28.422832 27805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:28.425132 27805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:28.425411 27805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:28.425670 27805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.430864 27813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.430871 27811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.430950 27805 server_base.cc:1034] running on GCE node
W20260927 07:04:28.430864 27810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.431293 27805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.431600 27805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.432742 27805 hybrid_clock.cc:648] HybridClock initialized: now 1790492668432732 us; error 30 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.435528 27805 init.cc:377] Logged in from keytab as oryx/127.21.2.66@KRBTEST.COM (short username oryx)
I20260927 07:04:28.435911 27805 webserver.cc:460] Webserver started at http://127.21.2.66:37249/ using document root <none> and password file <none>
I20260927 07:04:28.436111 27805 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.436151 27805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.436239 27805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:28.437227 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "404f3a2773524e019ca10a01fc0e277d"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "63c54ada6cebf7e28ba06de35eef3a96"
server_key_iv: "06e6ede05530802fd06ccd09ba8ce7f1"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.437526 27805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "404f3a2773524e019ca10a01fc0e277d"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "63c54ada6cebf7e28ba06de35eef3a96"
server_key_iv: "06e6ede05530802fd06ccd09ba8ce7f1"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.438747 27805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:28.439419 27820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.439589 27805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:28.439671 27805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "404f3a2773524e019ca10a01fc0e277d"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "63c54ada6cebf7e28ba06de35eef3a96"
server_key_iv: "06e6ede05530802fd06ccd09ba8ce7f1"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.439739 27805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.452734 27805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.455086 27805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:28.455260 27805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.455528 27805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:28.455854 27805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:28.455888 27805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.455928 27805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:28.455955 27805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.461473 27805 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46287
I20260927 07:04:28.461544 27933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46287 every 8 connection(s)
I20260927 07:04:28.461864 27805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:04:28.464756 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27805
I20260927 07:04:28.464865 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:04:28.465049 21513 external_mini_cluster.cc:1428] Setting key 49ef60f046c1ddc8a18a47c974c510bc
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:04:28.472036 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.462597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:33633 (local address 127.21.2.126:45915)
0927 07:04:28.462658 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.462661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.462869 (+ 208us) server_negotiation.cc:407] Connection header received
0927 07:04:28.463350 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.463351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.463398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.463458 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.464330 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.464835 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.466077 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.466277 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.466352 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.469131 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.469132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.469132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.469139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.470507 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.471016 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.471017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.471017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.471076 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.471406 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.471407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.471408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.471515 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.471567 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.471904 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.471972 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 07:04:28.472393 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.462580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:33633)
0927 07:04:28.462750 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.462784 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.463266 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.463511 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.463514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.464074 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.464257 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.464261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.464955 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.464956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.465996 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.465999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.466361 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.466362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.466496 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.467080 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.467091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.469030 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.470589 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.470597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.470600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.470939 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.471145 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.471146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.471146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.471300 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.471580 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.471585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.471756 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.471879 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.472049 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260927 07:04:28.472867 27934 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:28.472977 27934 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:28.473146 27934 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:28.473548 27609 ts_manager.cc:194] Registered new tserver with Master: 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287)
I20260927 07:04:28.473915 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:33633
WARNING: no policy specified for oryx/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
I20260927 07:04:28.490319 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:28.573865 27939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:28.574093 27939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:28.574132 27939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:28.575341 27939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:28.575429 27939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:28.575477 27939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:28.576972 27939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:28.577214 27939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:28.577468 27939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.582482 27944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.582496 27947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.582630 27939 server_base.cc:1034] running on GCE node
W20260927 07:04:28.582496 27945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.582943 27939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.583189 27939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.584362 27939 hybrid_clock.cc:648] HybridClock initialized: now 1790492668584328 us; error 65 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.586805 27939 init.cc:377] Logged in from keytab as oryx/127.21.2.67@KRBTEST.COM (short username oryx)
I20260927 07:04:28.587220 27939 webserver.cc:460] Webserver started at http://127.21.2.67:42487/ using document root <none> and password file <none>
I20260927 07:04:28.587473 27939 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.587519 27939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.587617 27939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:28.588575 27939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "8eb57ed582564f78bada13ceec7368bd"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "4f5a9ae5b172fc2cf7a067ab5b17f40c"
server_key_iv: "d28df4e3e13f4174f762b431628ac954"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.588857 27939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "8eb57ed582564f78bada13ceec7368bd"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "4f5a9ae5b172fc2cf7a067ab5b17f40c"
server_key_iv: "d28df4e3e13f4174f762b431628ac954"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.590057 27939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:28.590713 27954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.590859 27939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:28.590950 27939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "8eb57ed582564f78bada13ceec7368bd"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "4f5a9ae5b172fc2cf7a067ab5b17f40c"
server_key_iv: "d28df4e3e13f4174f762b431628ac954"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.591005 27939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.603030 27939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.605098 27939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:28.605249 27939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.605474 27939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:28.605783 27939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:28.605816 27939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.605858 27939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:28.605885 27939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.611951 27939 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:36053
I20260927 07:04:28.611985 28067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:36053 every 8 connection(s)
I20260927 07:04:28.612326 27939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:04:28.615046 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 27939
I20260927 07:04:28.615147 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:04:28.615337 21513 external_mini_cluster.cc:1428] Setting key 6570b0cf9b58d606dd8a4d81713dde26
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:04:28.621802 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.613124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:56333 (local address 127.21.2.126:45915)
0927 07:04:28.613171 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.613173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.613389 (+ 216us) server_negotiation.cc:407] Connection header received
0927 07:04:28.613870 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.613871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.613882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.613936 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.614880 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.615660 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.616877 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.617071 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.617127 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.619208 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.619210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.619210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.619217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.620433 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.620859 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.620861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.620861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.620922 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.621167 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.621168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.621168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.621284 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.621355 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.621625 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.621704 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:04:28.622027 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.613087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:56333)
0927 07:04:28.613265 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.613308 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.613776 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.614012 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.614015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.614587 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.614790 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.614794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.615759 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.615760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.616807 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.616810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.617144 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.617145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.617245 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.617734 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.617742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.619100 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.620539 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.620546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.620548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.620772 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.620986 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.620987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.620988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.621097 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.621346 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.621351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.621495 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.621604 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.621769 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260927 07:04:28.622510 28068 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:28.622601 28068 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:28.622742 28068 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:28.623063 27609 ts_manager.cc:194] Registered new tserver with Master: 8eb57ed582564f78bada13ceec7368bd (127.21.2.67:36053)
I20260927 07:04:28.623443 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:56333
I20260927 07:04:28.627115 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.125.keytab.
Entry for principal oryx/127.21.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.125.keytab.
I20260927 07:04:28.654428 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:44679
--webserver_interface=127.21.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.125.keytab
--principal=oryx/127.21.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45915,127.21.2.125:44679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:28.740229 28073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:28.740449 28073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:28.740486 28073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:28.741663 28073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:28.741719 28073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:28.741745 28073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:28.741775 28073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:28.741803 28073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:28.743338 28073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45915,127.21.2.125:44679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.125.keytab
--principal=oryx/127.21.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
W20260927 07:04:28.743489 28073 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.2.126:45915,127.21.2.125:44679'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 07:04:28.743645 28073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:28.743860 28073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.748544 28081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.748519 28078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.748541 28079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.748782 28073 server_base.cc:1034] running on GCE node
I20260927 07:04:28.749011 28073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.749295 28073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.750460 28073 hybrid_clock.cc:648] HybridClock initialized: now 1790492668750459 us; error 35 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.753504 28073 init.cc:377] Logged in from keytab as oryx/127.21.2.125@KRBTEST.COM (short username oryx)
I20260927 07:04:28.753949 28073 webserver.cc:460] Webserver started at http://127.21.2.125:33065/ using document root <none> and password file <none>
I20260927 07:04:28.754187 28073 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.754251 28073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.754396 28073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:28.755548 28073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/instance:
uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "a3083ea51c9850ddb7e98e369d5b7ecb"
server_key_iv: "300ac9ff5f84e71fa5a337c23d66ef6b"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.755899 28073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal/instance:
uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "a3083ea51c9850ddb7e98e369d5b7ecb"
server_key_iv: "300ac9ff5f84e71fa5a337c23d66ef6b"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.757268 28073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:04:28.758073 28088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.758281 28073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:28.758373 28073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "a3083ea51c9850ddb7e98e369d5b7ecb"
server_key_iv: "300ac9ff5f84e71fa5a337c23d66ef6b"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.758467 28073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.779088 28073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.780970 28073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:28.781123 28073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.785059 28073 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.125:44679
I20260927 07:04:28.785104 28140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.125:44679 every 8 connection(s)
I20260927 07:04:28.785452 28073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
I20260927 07:04:28.786396 28141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:28.787945 28141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:04:28.789865 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 28073
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.125@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:28.797861 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.788561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.21.2.126:45915)
0927 07:04:28.788620 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.788622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.788806 (+ 184us) server_negotiation.cc:407] Connection header received
0927 07:04:28.789239 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.789240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.789250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.789287 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.789850 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.790535 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.791750 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.791935 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.791991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.794446 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.794448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.794448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.794455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.796143 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.796640 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.796641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.796641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.796706 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.797125 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.797126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.797126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.797329 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.797400 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.797734 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.797796 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:04:28.798087 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.788538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38776)
0927 07:04:28.788695 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.788728 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.789152 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.789344 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.789347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.789586 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.789768 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.789772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.790623 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.790624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.791673 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.791675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.792010 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.792011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.792148 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.792690 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.792697 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.794356 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.796238 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.796244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.796246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.796572 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.796894 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.796894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.796895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.797059 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.797399 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.797405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.797589 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.797714 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.797856 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260927 07:04:28.798846 28141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:04:28.799727 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.790359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.21.2.125:44679)
0927 07:04:28.790643 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.790647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.790666 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:04:28.790685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.790687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.790706 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.790782 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.791358 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.792342 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.793176 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.793533 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.793602 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.796266 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.796271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.796271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.796278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.798256 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.798682 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.798685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.798686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.798748 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.798998 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.798999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.798999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.799188 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.799420 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.799478 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.799603 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":78,"threads_started":1}
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.125@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:28.805356 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.799078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56770)
0927 07:04:28.799122 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.799131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.799164 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.799416 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.799417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.799628 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.799681 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.799682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.800478 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.800479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.800999 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.801001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.801369 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.801370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.801433 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.801758 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.801762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.802801 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.804613 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.804614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.804614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.804768 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.804913 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.804914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.804914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.804969 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.805129 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.805130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.805161 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.805209 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.805276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 07:04:28.805473 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.799132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.21.2.125:44679)
0927 07:04:28.799271 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.799272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.799278 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:04:28.799291 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.799291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.799298 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.799365 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.799754 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.800369 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.801081 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.801296 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.801370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.802879 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.802880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.802880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.802885 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.804549 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.804824 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.804825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.804825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.804865 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.805006 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.805007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.805007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.805085 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.805128 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.805223 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.805317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260927 07:04:28.806630 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Bootstrap starting.
I20260927 07:04:28.807220 28141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:28.807533 28141 log.cc:824] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:28.808197 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: No bootstrap required, opened a new log
I20260927 07:04:28.809626 28141 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } }
I20260927 07:04:28.809788 28141 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:28.809855 28141 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e64e84611f4e77ac9ed15d727fba9c, State: Initialized, Role: FOLLOWER
I20260927 07:04:28.809970 28141 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } }
I20260927 07:04:28.810242 28141 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:28.810204 28148 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } } }
I20260927 07:04:28.810312 28148 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:04:28.811753 28159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:28.811813 28159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:28.813267 28073 master_runner.cc:428] Detected that this master b2e64e84611f4e77ac9ed15d727fba9c is joining an existing cluster
I20260927 07:04:28.813318 28073 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.2.125:44679
I20260927 07:04:28.820436 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.814236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.21.2.126:45915)
0927 07:04:28.814275 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.814277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.814520 (+ 243us) server_negotiation.cc:407] Connection header received
0927 07:04:28.814591 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.814592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.814602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.814666 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.815034 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.815566 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.816381 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.816574 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.816651 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.817701 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.817703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.817703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.817709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.819184 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.819660 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.819661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.819662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.819718 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.819908 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.819909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.819910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.820028 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.820102 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.820265 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.820339 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:04:28.820472 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.814207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38784)
0927 07:04:28.814454 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.814464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.814532 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.814670 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.814670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.814884 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.814951 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.814953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.815637 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.815638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.816303 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.816305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.816671 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.816672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.816734 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.817132 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.817137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.817435 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.819272 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.819273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.819273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.819570 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.819787 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.819788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.819789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.819847 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.820093 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.820094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.820136 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.820227 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.820361 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260927 07:04:28.821509 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.814259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.21.2.125:44679)
0927 07:04:28.814292 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.814294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.814524 (+ 230us) server_negotiation.cc:407] Connection header received
0927 07:04:28.814606 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.814606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.814614 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.814639 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.815077 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.815814 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.816906 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.817227 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.817263 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.818159 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.818160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.818161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.818166 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.820052 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.820475 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.820475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.820476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.820542 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.820716 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.820717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.820718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.820847 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.820914 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.821347 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.821426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:28.821858 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.814337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56776)
0927 07:04:28.814466 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.814478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.814545 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.814688 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.814688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.814920 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.814995 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.814997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.816003 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.816004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.816816 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.816819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.817326 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.817327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.817379 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.817699 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.817704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.818071 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.820136 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.820137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.820137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.820397 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.820601 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.820601 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.820601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.820648 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.820920 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.820921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.820971 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.821622 (+ 651us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.821718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.126@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:28.829895 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.823040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56784)
0927 07:04:28.823163 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.823202 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.823269 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.823419 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.823423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.823672 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.823716 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.823718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.824566 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.824567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.825124 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.825125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.825565 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.825566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.825618 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.826051 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.826055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.827294 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.828941 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.828948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.828950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.829136 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.829326 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.829327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.829327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.829379 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.829624 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.829629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.829671 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.829731 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.829816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260927 07:04:28.829941 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.823096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.21.2.125:44679)
0927 07:04:28.823130 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.823132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.823267 (+ 135us) server_negotiation.cc:407] Connection header received
0927 07:04:28.823322 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.823322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.823331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.823412 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.823788 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.824494 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.825194 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.825487 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.825550 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.827401 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.827402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.827403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.827409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.828853 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.829201 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.829202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.829202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.829260 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.829434 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.829435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.829435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.829558 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.829618 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.829782 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.829852 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:28.830425 27609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.2.125:44679: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
member_type: NON_VOTER
last_known_addr {
host: "127.21.2.125"
port: 44679
}
attrs {
promote: true
}
}
dest_uuid: "287e9eeb51094b37844e0a0ad282e4ec"
cas_config_opid_index: -1
I20260927 07:04:28.830742 27609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } }
I20260927 07:04:28.831171 28169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b2e64e84611f4e77ac9ed15d727fba9c (127.21.2.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } }
I20260927 07:04:28.831471 28115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:28.832160 28169 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.2.125:44679
I20260927 07:04:28.832345 28115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Refusing update from remote peer 287e9eeb51094b37844e0a0ad282e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 07:04:28.832409 28073 master.cc:537] Master@127.21.2.125:44679 shutting down...
I20260927 07:04:28.832661 28168 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:28.832918 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } }
W20260927 07:04:28.833446 27596 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 07:04:28.833703 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
W20260927 07:04:28.834098 27596 consensus_peers.cc:487] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec -> Peer b2e64e84611f4e77ac9ed15d727fba9c (127.21.2.125:44679): Couldn't send request to peer b2e64e84611f4e77ac9ed15d727fba9c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 07:04:28.835034 28073 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:28.835116 28073 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:28.835180 28073 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: stopping tablet replica
I20260927 07:04:28.847725 28073 master.cc:552] Master@127.21.2.125:44679 shutdown complete.
I20260927 07:04:28.847769 28073 master_runner.cc:305] Clearing existing system tablet
I20260927 07:04:28.847909 28073 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.847949 28073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.848548 28073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:28.849014 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.849125 28073 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.849184 28073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.849206 28073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "a3083ea51c9850ddb7e98e369d5b7ecb"
server_key_iv: "300ac9ff5f84e71fa5a337c23d66ef6b"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.849826 28073 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:28.851328 28073 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 07:04:28.851543 28073 log.cc:1196] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 07:04:28.851769 28073 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Deleting consensus metadata
I20260927 07:04:28.851894 28073 master_runner.cc:315] Copying system tablet from 127.21.2.126:45915
I20260927 07:04:28.852829 28073 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.126:45915
I20260927 07:04:28.859139 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.853250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.21.2.126:45915)
0927 07:04:28.853298 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.853300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.853427 (+ 127us) server_negotiation.cc:407] Connection header received
0927 07:04:28.853518 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.853519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.853530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.853585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.854142 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.854637 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.855356 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.855632 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.855705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.856680 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.856681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.856682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.856689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.858063 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.858386 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.858387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.858388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.858449 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.858638 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.858639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.858639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.858752 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.858823 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.858965 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.859048 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:28.859149 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.853188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38796)
0927 07:04:28.853356 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:28.853366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:28.853454 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:28.853591 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:28.853592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:28.853765 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:28.854053 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.854055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.854717 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.854718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.855273 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:28.855274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.855717 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.855719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:28.855785 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.856274 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:28.856279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:28.856589 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:28.858143 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.858144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.858144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.858317 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.858511 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:28.858512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:28.858512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:28.858575 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:28.858815 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:28.858815 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:28.858860 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:04:28.858942 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:04:28.859043 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260927 07:04:28.859489 27629 tablet_copy_service.cc:140] P 287e9eeb51094b37844e0a0ad282e4ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b2e64e84611f4e77ac9ed15d727fba9c ({username='oryx', principal='oryx/127.21.2.125@KRBTEST.COM'} at 127.0.0.1:38796)
I20260927 07:04:28.859593 27629 tablet_copy_service.cc:161] P 287e9eeb51094b37844e0a0ad282e4ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b2e64e84611f4e77ac9ed15d727fba9c at {username='oryx', principal='oryx/127.21.2.125@KRBTEST.COM'} at 127.0.0.1:38796: session id = b2e64e84611f4e77ac9ed15d727fba9c-00000000000000000000000000000000
I20260927 07:04:28.860237 27629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 07:04:28.860836 28073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 07:04:28.861754 28073 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: tablet copy: Starting download of 0 data blocks...
I20260927 07:04:28.861900 28073 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: tablet copy: Starting download of 1 WAL segments...
I20260927 07:04:28.862718 28073 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 07:04:28.863538 27629 tablet_copy_service.cc:342] P 287e9eeb51094b37844e0a0ad282e4ec: Request end of tablet copy session b2e64e84611f4e77ac9ed15d727fba9c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.2.125@KRBTEST.COM'} at 127.0.0.1:38796
I20260927 07:04:28.863597 27629 tablet_copy_service.cc:433] P 287e9eeb51094b37844e0a0ad282e4ec: ending tablet copy session b2e64e84611f4e77ac9ed15d727fba9c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b2e64e84611f4e77ac9ed15d727fba9c
I20260927 07:04:28.866001 28073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:28.867190 28181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:28.867249 28184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.867313 28073 server_base.cc:1034] running on GCE node
W20260927 07:04:28.867443 28182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:28.867662 28073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:28.867808 28073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:28.868928 28073 hybrid_clock.cc:648] HybridClock initialized: now 1790492668868917 us; error 31 us; skew 500 ppm
Sep 27 07:04:28 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:28.870203 28073 init.cc:377] Logged in from keytab as oryx/127.21.2.125@KRBTEST.COM (short username oryx)
I20260927 07:04:28.870482 28073 webserver.cc:460] Webserver started at http://127.21.2.125:35259/ using document root <none> and password file <none>
I20260927 07:04:28.870608 28073 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:28.870646 28073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:28.871241 28073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:28.871788 28191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.871902 28073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:28.871949 28073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "b2e64e84611f4e77ac9ed15d727fba9c"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "a3083ea51c9850ddb7e98e369d5b7ecb"
server_key_iv: "300ac9ff5f84e71fa5a337c23d66ef6b"
server_key_version: "encryptionkey@0"
I20260927 07:04:28.872004 28073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:28.889582 28073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:28.891196 28073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:28.894410 28073 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.125:44679
I20260927 07:04:28.894505 28243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.125:44679 every 8 connection(s)
I20260927 07:04:28.894735 28073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
I20260927 07:04:28.895470 28244 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:04:28.895889 28244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Bootstrap starting.
I20260927 07:04:28.898047 28244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:04:28.898268 28244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: Bootstrap complete.
I20260927 07:04:28.898468 28244 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } }
I20260927 07:04:28.898536 28244 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e64e84611f4e77ac9ed15d727fba9c, State: Initialized, Role: LEARNER
I20260927 07:04:28.898563 28244 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } }
I20260927 07:04:28.898739 28244 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:28.898744 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } }
I20260927 07:04:28.898895 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: This master's current role is: LEARNER
I20260927 07:04:28.900206 28258 catalog_manager.cc:1259] Loaded cluster ID: bedb1b25816649b2ba279fd49d78ea2f
I20260927 07:04:28.900255 28258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: loading cluster ID for follower catalog manager: success
I20260927 07:04:28.901489 28258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: acquiring CA information for follower catalog manager: success
I20260927 07:04:28.901707 28258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:04:28.913100 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:28.901181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.21.2.125:44679)
0927 07:04:28.903469 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:28.903471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:28.903483 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:04:28.903508 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:28.903508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:28.903519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:28.903608 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:28.904115 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.905110 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.905790 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:28.906108 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:28.906190 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:28.910358 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:28.910360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:28.910360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:28.910368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:28.912004 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.912330 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.912332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.912332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.912397 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:28.912577 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:28.912578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:28.912578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:28.912718 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:28.912836 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:28.912905 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 07:04:28.912989 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":55,"threads_started":1}
WARNING: no policy specified for oryx/127.21.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.124.keytab.
Entry for principal oryx/127.21.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.124.keytab.
I20260927 07:04:28.936121 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:35019
--webserver_interface=127.21.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.124.keytab
--principal=oryx/127.21.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:04:29.018860 28262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:29.019075 28262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:29.019097 28262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:29.020298 28262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:04:29.020339 28262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:29.020352 28262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:29.020365 28262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:04:29.020377 28262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:04:29.021816 28262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.124.keytab
--principal=oryx/127.21.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:29.022003 28262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:29.022169 28262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:29.026778 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.027087 28262 server_base.cc:1034] running on GCE node
W20260927 07:04:29.026822 28268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.026778 28267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.027500 28262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:29.027783 28262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:29.028942 28262 hybrid_clock.cc:648] HybridClock initialized: now 1790492669028928 us; error 31 us; skew 500 ppm
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:29.031455 28262 init.cc:377] Logged in from keytab as oryx/127.21.2.124@KRBTEST.COM (short username oryx)
I20260927 07:04:29.031865 28262 webserver.cc:460] Webserver started at http://127.21.2.124:41997/ using document root <none> and password file <none>
I20260927 07:04:29.032043 28262 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.032083 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.032224 28262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:04:29.033236 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/instance:
uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
format_stamp: "Formatted at 2026-09-27 07:04:29 on dist-test-slave-ltqz"
server_key: "6d5f30d58788f0ef922b364d1351dc63"
server_key_iv: "b6c2367ef91ce0717ddd07f01e636f3a"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.033593 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal/instance:
uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
format_stamp: "Formatted at 2026-09-27 07:04:29 on dist-test-slave-ltqz"
server_key: "6d5f30d58788f0ef922b364d1351dc63"
server_key_iv: "b6c2367ef91ce0717ddd07f01e636f3a"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.034850 28262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:04:29.035530 28277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.035687 28262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.035755 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
format_stamp: "Formatted at 2026-09-27 07:04:29 on dist-test-slave-ltqz"
server_key: "6d5f30d58788f0ef922b364d1351dc63"
server_key_iv: "b6c2367ef91ce0717ddd07f01e636f3a"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.035805 28262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.042701 28262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:29.045228 28262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:29.045334 28262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:29.049225 28262 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.124:35019
I20260927 07:04:29.049274 28329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.124:35019 every 8 connection(s)
I20260927 07:04:29.049578 28262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
I20260927 07:04:29.050493 28330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:29.050522 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 28262
I20260927 07:04:29.053158 28330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.124@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:04:29.061362 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.051288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.21.2.124:35019)
0927 07:04:29.051523 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.051531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.051556 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:04:29.052002 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.052008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.052054 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.052205 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.052820 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.053816 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.054667 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.055204 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.055280 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.057927 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.057933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.057933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.057939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.059804 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.060153 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.060156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.060156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.060242 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.060456 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.060457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.060457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.060676 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.060809 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.060960 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.061091 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260927 07:04:29.061916 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.054014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38802)
0927 07:04:29.054170 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.054209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.054286 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.054485 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.054488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.054820 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.054898 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.054900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.055710 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.055712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.056446 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.056449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.056960 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.056961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.057035 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.057608 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.057618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.059100 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.060791 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.060797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.060799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.060992 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.061200 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.061201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.061201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.061265 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.061550 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.061554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.061602 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.061697 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.061809 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260927 07:04:29.061908 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.054196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.21.2.126:45915)
0927 07:04:29.054253 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.054256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.054266 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:29.054349 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.054351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.054362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.054430 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.054983 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.055618 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.056541 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.056859 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.056926 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.059190 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.059191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.059192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.059200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.060690 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.061065 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.061067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.061067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.061131 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.061331 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.061332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.061332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.061465 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.061551 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.061732 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.061821 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:04:29.062400 28330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.124@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:29.068269 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.062645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56810)
0927 07:04:29.062687 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.062698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.062732 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.062863 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.062863 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.062972 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.063020 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.063021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.063680 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.063681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.064164 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.064165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.064481 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.064481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.064530 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.064841 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.064845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.065958 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.067546 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.067547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.067547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.067693 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.067850 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.067850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.067851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.067896 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.068066 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.068067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.068097 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.068145 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.068209 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 07:04:29.068289 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.062711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.21.2.125:44679)
0927 07:04:29.062751 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.062753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.062776 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:04:29.062807 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.062807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.062819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.062854 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.063067 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.063618 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.064226 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.064407 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.064455 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.066033 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.066035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.066035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.066041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.067468 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.067748 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.067748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.067749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.067796 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.067933 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.067933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.067933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.068021 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.068067 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.068161 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.068227 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:29.068708 28330 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 35019 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.124@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
I20260927 07:04:29.074370 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.068947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.0.0.1:59440)
0927 07:04:29.069032 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.069041 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.069077 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.069167 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.069168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.069268 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.069295 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.069296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.069841 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.069842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.070288 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.070289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.070605 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.070606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.070639 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.070999 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.071003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.072095 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.073534 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.073535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.073536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.073704 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.073875 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.073876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.073876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.073927 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.074115 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.074115 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.074147 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.074197 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.074281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":2}
I20260927 07:04:29.074440 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.068994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.21.2.124:35019)
0927 07:04:29.069043 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.069045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.069067 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:04:29.069111 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.069112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.069119 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.069159 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.069333 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.069777 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.070351 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.070548 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.070597 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.072163 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.072164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.072165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.072171 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.073448 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.073769 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.073769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.073770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.073815 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.073975 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.073976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.073976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.074066 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.074113 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.074210 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.074322 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":34}
I20260927 07:04:29.075834 28330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Bootstrap starting.
I20260927 07:04:29.076591 28330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:29.076812 28330 log.cc:824] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:29.077448 28330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: No bootstrap required, opened a new log
I20260927 07:04:29.078860 28330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 35019 } }
I20260927 07:04:29.078977 28330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:29.079006 28330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543c9ad1aa334e70a6f831d4cbf094fa, State: Initialized, Role: FOLLOWER
I20260927 07:04:29.079090 28330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 35019 } }
I20260927 07:04:29.079327 28335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 35019 } } }
I20260927 07:04:29.079412 28330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:29.079432 28335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:04:29.081077 28346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:04:29.081161 28346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:04:29.083300 28262 master_runner.cc:428] Detected that this master 543c9ad1aa334e70a6f831d4cbf094fa is joining an existing cluster
I20260927 07:04:29.083346 28262 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.2.124:35019
I20260927 07:04:29.091295 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.21.2.124:35019)
0927 07:04:29.084563 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.084565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.084900 (+ 335us) server_negotiation.cc:407] Connection header received
0927 07:04:29.084982 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.084983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.084991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.085017 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.085565 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086269 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087332 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087630 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087673 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.088765 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.088766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.088767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.088772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.090149 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.090562 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.090562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.090563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.090649 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.090850 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.090851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.090851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.090948 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.090996 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.091167 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.091233 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:04:29.091727 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.0.0.1:59450)
0927 07:04:29.084839 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.084848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.084920 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.085069 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.085070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.085404 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.085476 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.085478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086358 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.086359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087240 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.087243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087752 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087812 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.088231 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.088237 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.088672 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.090232 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.090233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.090233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.090483 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.090721 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.090722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.090722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.090788 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.091009 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.091010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.091055 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.091143 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.091303 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":50,"thread_start_us":94,"threads_started":1}
I20260927 07:04:29.112910 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27671
I20260927 07:04:29.116097 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.21.2.126:45915)
0927 07:04:29.084560 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.084562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.084635 (+ 73us) server_negotiation.cc:407] Connection header received
0927 07:04:29.084759 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.084760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.084773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.084834 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.085348 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086144 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087318 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087644 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087686 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.112856 (+ 25170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.112859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.112859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.112869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.114724 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115141 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.115143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.115143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.115196 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115434 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.115435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.115436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.115604 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.115650 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.115858 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.115947 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:04:29.116338 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.21.2.125:44679)
0927 07:04:29.084754 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.084756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.084904 (+ 148us) server_negotiation.cc:407] Connection header received
0927 07:04:29.085015 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.085016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.085026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.085057 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.085529 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086243 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087318 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087645 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087686 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.112957 (+ 25271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.112959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.112959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.112968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.115126 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115541 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.115542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.115542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.115597 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115786 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.115786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.115787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.115925 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.115968 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.116165 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.116247 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:29.116647 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56818)
0927 07:04:29.084846 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.084856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.084928 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.085122 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.085123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.085393 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.085448 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.085450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086331 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.086331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087220 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.087221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087767 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087839 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.112125 (+ 24286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.112131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.112823 (+ 692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.115227 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.115228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.115453 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.115667 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.115668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.115720 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.116014 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.116015 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.116055 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.116422 (+ 367us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.116535 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":39,"thread_start_us":72,"threads_started":1}
I20260927 07:04:29.116917 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.084392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38808)
0927 07:04:29.084574 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.084585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.084646 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.084887 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.084888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.085168 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.085252 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.085254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.086233 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.086234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087216 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.087219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.087767 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.087767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.087839 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.112199 (+ 24360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.112204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.112742 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.114822 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.114823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.114823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.115058 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.115269 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.115270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.115270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.115331 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.115696 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.115697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.115747 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.116730 (+ 983us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.116815 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":61,"thread_start_us":109,"threads_started":1}
I20260927 07:04:29.120911 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:34707
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:45915
--webserver_port=45979
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492668, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.126@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
I20260927 07:04:29.127187 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.119127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.0.0.1:59458)
0927 07:04:29.119184 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.119196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.119240 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.119531 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.119532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.119699 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.119748 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.119749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.120644 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.120645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.121475 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.121477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.121955 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.121956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.122022 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.122618 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.122624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.124153 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.126000 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.126001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.126002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.126213 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.126457 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.126458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.126458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.126525 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.126823 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.126824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.126879 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.126953 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.127059 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:04:29.127214 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.119318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.21.2.124:35019)
0927 07:04:29.119410 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.119412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.119419 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:04:29.119451 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.119452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.119462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.119511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.119820 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.120570 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.121561 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.121876 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.121950 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.124251 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.124252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.124253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.124259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.125905 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.126293 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.126295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.126295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.126362 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.126595 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.126597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.126598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.126749 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.126827 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.126995 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.127080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:04:29.128163 27609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.2.124:35019: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
member_type: NON_VOTER
last_known_addr {
host: "127.21.2.124"
port: 35019
}
attrs {
promote: true
}
}
dest_uuid: "287e9eeb51094b37844e0a0ad282e4ec"
cas_config_opid_index: 5
I20260927 07:04:29.128391 27609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:29.128724 28170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 543c9ad1aa334e70a6f831d4cbf094fa (127.21.2.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.129765 28170 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.2.124:35019
I20260927 07:04:29.129843 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.129937 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:29.130410 28262 master.cc:537] Master@127.21.2.124:35019 shutting down...
I20260927 07:04:29.130489 28304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:29.132081 28304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 1 FOLLOWER]: Refusing update from remote peer 287e9eeb51094b37844e0a0ad282e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 07:04:29.132488 28169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 07:04:29.133126 27595 consensus_peers.cc:487] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec -> Peer 543c9ad1aa334e70a6f831d4cbf094fa (127.21.2.124:35019): Couldn't send request to peer 543c9ad1aa334e70a6f831d4cbf094fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 07:04:29.134763 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.130134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.0.0.1:56820)
0927 07:04:29.130898 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.130910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.130963 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.131201 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.131201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.131203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.131242 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.131243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.132210 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.132211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.133582 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.133585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.134506 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.134506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.134583 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.134604 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.134651 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.134698 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746}
I20260927 07:04:29.135051 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.130216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.21.2.125:44679)
0927 07:04:29.130262 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.130265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.131020 (+ 755us) server_negotiation.cc:407] Connection header received
0927 07:04:29.131049 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.131050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.131088 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.131144 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.131308 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.132113 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.133682 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.134301 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.134352 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.134429 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.134922 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.134969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:29.135275 28218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 LEARNER]: Refusing update from remote peer 287e9eeb51094b37844e0a0ad282e4ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 07:04:29.135514 28169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 07:04:29.136070 28262 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:29.136143 28262 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:29.136193 28262 tablet_replica.cc:330] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: stopping tablet replica
I20260927 07:04:29.136060 28218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 543c9ad1aa334e70a6f831d4cbf094fa (127.21.2.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.137315 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287e9eeb51094b37844e0a0ad282e4ec. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.137413 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: This master's current role is: LEARNER
I20260927 07:04:29.137449 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: NON_VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: true } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.137488 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: This master's current role is: LEARNER
I20260927 07:04:29.137694 28169 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: attempting to promote NON_VOTER b2e64e84611f4e77ac9ed15d727fba9c to VOTER
I20260927 07:04:29.137868 28169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:29.138412 28218 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 LEARNER]: Refusing update from remote peer 287e9eeb51094b37844e0a0ad282e4ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 07:04:29.138594 28170 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 07:04:29.138887 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.138313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.0.0.1:59464)
0927 07:04:29.138803 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.138844 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:35019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
I20260927 07:04:29.139287 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.139071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.0.0.1:59466)
0927 07:04:29.139239 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.139261 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:35019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
W20260927 07:04:29.139336 27595 consensus_peers.cc:487] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec -> Peer 543c9ad1aa334e70a6f831d4cbf094fa (127.21.2.124:35019): Couldn't send request to peer 543c9ad1aa334e70a6f831d4cbf094fa. Status: Network error: Client connection negotiation failed: client connection to 127.21.2.124:35019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 07:04:29.139339 28170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e64e84611f4e77ac9ed15d727fba9c (127.21.2.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.139680 28218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e64e84611f4e77ac9ed15d727fba9c (127.21.2.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.140205 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.140270 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:29.140316 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.140352 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec [sys.catalog]: This master's current role is: LEADER
I20260927 07:04:29.140522 28247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "287e9eeb51094b37844e0a0ad282e4ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.140594 28247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:04:29.149690 28262 master.cc:552] Master@127.21.2.124:35019 shutdown complete.
I20260927 07:04:29.149741 28262 master_runner.cc:305] Clearing existing system tablet
I20260927 07:04:29.149863 28262 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.149907 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.150630 28262 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:04:29.151510 28360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.151693 28262 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.151759 28262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:29.151785 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
format_stamp: "Formatted at 2026-09-27 07:04:29 on dist-test-slave-ltqz"
server_key: "6d5f30d58788f0ef922b364d1351dc63"
server_key_iv: "b6c2367ef91ce0717ddd07f01e636f3a"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.152581 28262 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:29.154074 28262 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 07:04:29.154145 28262 log.cc:1196] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 07:04:29.154327 28262 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Deleting consensus metadata
I20260927 07:04:29.154477 28262 master_runner.cc:315] Copying system tablet from 127.21.2.126:45915
I20260927 07:04:29.155158 28262 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.126:45915
I20260927 07:04:29.162457 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.155551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.21.2.126:45915)
0927 07:04:29.155599 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.155602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.155698 (+ 96us) server_negotiation.cc:407] Connection header received
0927 07:04:29.155759 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.155760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.155772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.155871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.156213 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.156994 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.158016 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.158307 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.158391 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.159594 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.159596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.159597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.159605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.161256 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.161646 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.161647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.161647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.161707 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.161904 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.161905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.161906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.162047 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.162119 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.162290 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.162367 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:29.162478 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.155491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38810)
0927 07:04:29.155626 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.155637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.155702 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.155877 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.155878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.156077 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.156124 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.156126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.157090 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.157091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.157913 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.157917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.158402 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.158403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.158564 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.159050 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.159056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.159507 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.161342 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.161344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.161344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.161569 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.161775 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.161776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.161776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.161841 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.162117 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.162118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.162170 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.162249 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.162366 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260927 07:04:29.163040 27629 tablet_copy_service.cc:140] P 287e9eeb51094b37844e0a0ad282e4ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 543c9ad1aa334e70a6f831d4cbf094fa ({username='oryx', principal='oryx/127.21.2.124@KRBTEST.COM'} at 127.0.0.1:38810)
I20260927 07:04:29.163108 27629 tablet_copy_service.cc:161] P 287e9eeb51094b37844e0a0ad282e4ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 543c9ad1aa334e70a6f831d4cbf094fa at {username='oryx', principal='oryx/127.21.2.124@KRBTEST.COM'} at 127.0.0.1:38810: session id = 543c9ad1aa334e70a6f831d4cbf094fa-00000000000000000000000000000000
I20260927 07:04:29.163725 27629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 07:04:29.164199 28262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 07:04:29.165503 28262 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: tablet copy: Starting download of 0 data blocks...
I20260927 07:04:29.165648 28262 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: tablet copy: Starting download of 1 WAL segments...
I20260927 07:04:29.166383 28262 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 07:04:29.167302 27629 tablet_copy_service.cc:342] P 287e9eeb51094b37844e0a0ad282e4ec: Request end of tablet copy session 543c9ad1aa334e70a6f831d4cbf094fa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.2.124@KRBTEST.COM'} at 127.0.0.1:38810
I20260927 07:04:29.167356 27629 tablet_copy_service.cc:433] P 287e9eeb51094b37844e0a0ad282e4ec: ending tablet copy session 543c9ad1aa334e70a6f831d4cbf094fa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 543c9ad1aa334e70a6f831d4cbf094fa
I20260927 07:04:29.170413 28262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:29.171532 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.172470 28370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.172739 28262 server_base.cc:1034] running on GCE node
W20260927 07:04:29.175742 28368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.175920 28262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:29.176059 28262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:29.177179 28262 hybrid_clock.cc:648] HybridClock initialized: now 1790492669177161 us; error 37 us; skew 500 ppm
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:29.178719 28262 init.cc:377] Logged in from keytab as oryx/127.21.2.124@KRBTEST.COM (short username oryx)
I20260927 07:04:29.178982 28262 webserver.cc:460] Webserver started at http://127.21.2.124:37565/ using document root <none> and password file <none>
I20260927 07:04:29.179116 28262 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.179155 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.179924 28262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:04:29.180524 28377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.180670 28262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:04:29.180722 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "543c9ad1aa334e70a6f831d4cbf094fa"
format_stamp: "Formatted at 2026-09-27 07:04:29 on dist-test-slave-ltqz"
server_key: "6d5f30d58788f0ef922b364d1351dc63"
server_key_iv: "b6c2367ef91ce0717ddd07f01e636f3a"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.180776 28262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.196902 28262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:29.198843 28262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:29.201902 28262 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.124:35019
I20260927 07:04:29.202324 28262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
I20260927 07:04:29.203492 28430 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:04:29.204049 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Bootstrap starting.
I20260927 07:04:29.206772 28429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.124:35019 every 8 connection(s)
I20260927 07:04:29.207158 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:04:29.207417 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: Bootstrap complete.
I20260927 07:04:29.207625 28430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:29.207718 28430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 543c9ad1aa334e70a6f831d4cbf094fa, State: Initialized, Role: LEARNER
I20260927 07:04:29.207756 28430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:29.207998 28430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:04:29.207999 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } } }
I20260927 07:04:29.208269 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa [sys.catalog]: This master's current role is: LEARNER
I20260927 07:04:29.209646 28444 catalog_manager.cc:1259] Loaded cluster ID: bedb1b25816649b2ba279fd49d78ea2f
I20260927 07:04:29.209693 28444 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: loading cluster ID for follower catalog manager: success
I20260927 07:04:29.210834 28444 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: acquiring CA information for follower catalog manager: success
I20260927 07:04:29.211144 28444 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 543c9ad1aa334e70a6f831d4cbf094fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 07:04:29.224076 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:29.224259 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:29.224282 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:29.225423 28355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:29.225464 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:29.225490 28355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:04:29.227020 28355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:34707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=45979
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:29.227257 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:29.227517 28355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:29.232383 28449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.232399 28450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.232398 28452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.232837 28355 server_base.cc:1034] running on GCE node
I20260927 07:04:29.233040 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:29.233306 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:29.234468 28355 hybrid_clock.cc:648] HybridClock initialized: now 1790492669234457 us; error 30 us; skew 500 ppm
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:29.237051 28355 init.cc:377] Logged in from keytab as oryx/127.21.2.65@KRBTEST.COM (short username oryx)
I20260927 07:04:29.237550 28355 webserver.cc:460] Webserver started at http://127.21.2.65:45979/ using document root <none> and password file <none>
I20260927 07:04:29.237775 28355 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.237849 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.239172 28355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:04:29.239810 28459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.239959 28355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.240041 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "435025c5658f4879931d3d90a806516e"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "200386eb6cb9473fc5a945c4e0ae91a1"
server_key_iv: "a30f120c22ed3b435960e4cc260f6404"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.240381 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.262189 28355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:29.264189 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:29.264343 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:29.264592 28355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:29.264904 28355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:29.264937 28355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.264978 28355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:29.265005 28355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.270702 28355 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:34707
I20260927 07:04:29.270753 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:34707 every 8 connection(s)
I20260927 07:04:29.271140 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:04:29.276636 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 28355
I20260927 07:04:29.276800 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27805
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:29.283216 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:46287
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:45915
--webserver_port=37249
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:04:29.283491 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.271957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:53507 (local address 127.21.2.126:45915)
0927 07:04:29.272008 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.272010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.272364 (+ 354us) server_negotiation.cc:407] Connection header received
0927 07:04:29.273043 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.273044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.273057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.273109 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.273966 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.274731 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.275835 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276118 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276185 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.279840 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.279841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.279842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.279851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.281666 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.282255 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.282258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.282258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.282319 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.282648 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.282649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.282650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.282798 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.282842 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.283256 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.283338 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:29.284010 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.271896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.21.2.65:54533)
0927 07:04:29.272459 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.272467 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.272864 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.273262 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.273263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.273627 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.273851 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.273856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.275018 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.275019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.275711 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.275713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276223 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.276327 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.278033 (+ 1706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.278039 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.280496 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.282592 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.282593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.282593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.282810 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.283011 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.283012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.283074 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.283402 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.283403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.283459 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.283540 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.283731 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":183,"thread_start_us":63,"threads_started":1}
I20260927 07:04:29.284360 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.272136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:36693 (local address 127.21.2.125:44679)
0927 07:04:29.272183 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.272186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.272399 (+ 213us) server_negotiation.cc:407] Connection header received
0927 07:04:29.272957 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.272958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.272971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.273032 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.273963 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.274454 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.275863 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276070 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276157 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.281207 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.281208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.281209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.281217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.283105 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283518 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.283519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.283519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.283578 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283812 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.283812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.283813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.283941 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.283994 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.284188 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.284273 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:29.284384 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.271988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54533 (local address 127.21.2.124:35019)
0927 07:04:29.272156 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.272158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.272526 (+ 368us) server_negotiation.cc:407] Connection header received
0927 07:04:29.273168 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.273170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.273187 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.273251 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.273972 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.274931 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.275793 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276107 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276174 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.280596 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.280598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.280598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.280607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.282506 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.282883 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.282884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.282885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.282940 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283141 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.283142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.283142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.283278 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.283343 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.284139 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.284237 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":59,"threads_started":1}
I20260927 07:04:29.284691 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.271896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.21.2.65:36693)
0927 07:04:29.272224 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.272267 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.272855 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.273099 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.273104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.273665 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.273849 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.273854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.274559 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.274560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.275722 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.275724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276187 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.276327 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.278353 (+ 2026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.278357 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.280621 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.283195 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.283196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.283431 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.283664 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.283665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.283665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.283738 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.284027 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.284028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.284071 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.284464 (+ 393us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.284571 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260927 07:04:29.284996 28573 heartbeater.cc:347] Connected to a master server at 127.21.2.124:35019
I20260927 07:04:29.285018 28575 heartbeater.cc:347] Connected to a master server at 127.21.2.125:44679
I20260927 07:04:29.285133 28573 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.285135 28575 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.285224 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.271898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:53507)
0927 07:04:29.272229 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.272267 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.272855 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.273154 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.273155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.273634 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.273850 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.273856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.274811 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.274811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.275722 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.275724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.276230 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.276231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.276324 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.277420 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.277431 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.279740 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.281807 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.281814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.281816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.282150 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.282406 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.282406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.282407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.282553 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.282914 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.282920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.283095 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.284288 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.284393 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":44,"thread_start_us":76,"threads_started":1}
I20260927 07:04:29.285383 28575 heartbeater.cc:511] Master 127.21.2.125:44679 requested a full tablet report, sending...
I20260927 07:04:29.285372 28573 heartbeater.cc:511] Master 127.21.2.124:35019 requested a full tablet report, sending...
I20260927 07:04:29.285756 28574 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:29.285799 28574 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.285862 28574 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:29.285892 28394 ts_manager.cc:194] Registered new tserver with Master: 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
I20260927 07:04:29.285899 28208 ts_manager.cc:194] Registered new tserver with Master: 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
I20260927 07:04:29.286209 27609 ts_manager.cc:194] Re-registered known tserver with Master: 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
I20260927 07:04:29.286715 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:53507
W20260927 07:04:29.365157 28581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:29.365391 28581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:29.365428 28581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:29.366585 28581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:29.366652 28581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:29.366698 28581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:04:29.368228 28581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:46287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=37249
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:29.368484 28581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:29.368710 28581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:29.373679 28589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.373703 28587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.373807 28581 server_base.cc:1034] running on GCE node
W20260927 07:04:29.373674 28586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.374073 28581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:29.374321 28581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:29.375451 28581 hybrid_clock.cc:648] HybridClock initialized: now 1790492669375432 us; error 34 us; skew 500 ppm
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:29.377882 28581 init.cc:377] Logged in from keytab as oryx/127.21.2.66@KRBTEST.COM (short username oryx)
I20260927 07:04:29.378331 28581 webserver.cc:460] Webserver started at http://127.21.2.66:37249/ using document root <none> and password file <none>
I20260927 07:04:29.378572 28581 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.378651 28581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.380115 28581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:04:29.380858 28596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.381008 28581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:29.381107 28581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "404f3a2773524e019ca10a01fc0e277d"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "63c54ada6cebf7e28ba06de35eef3a96"
server_key_iv: "06e6ede05530802fd06ccd09ba8ce7f1"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.381495 28581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.390200 28581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:29.392388 28581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:29.392540 28581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:29.392763 28581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:29.393064 28581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:29.393096 28581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.393136 28581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:29.393162 28581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.398790 28581 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46287
I20260927 07:04:29.398859 28709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46287 every 8 connection(s)
I20260927 07:04:29.399235 28581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
I20260927 07:04:29.410116 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 28581
I20260927 07:04:29.410270 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27939
I20260927 07:04:29.411616 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.400231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:51421 (local address 127.21.2.126:45915)
0927 07:04:29.400282 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.400284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.400295 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:04:29.401336 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.401337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.401350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.401412 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.402185 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.402690 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.403716 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.403939 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404012 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.408129 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.408130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.408131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.408139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.409855 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.410390 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.410391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.410392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.410454 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.410706 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.410707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.410707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.410847 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.410890 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.411421 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.411508 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:29.411824 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.400387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:40949 (local address 127.21.2.124:35019)
0927 07:04:29.400427 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.400429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.400504 (+ 75us) server_negotiation.cc:407] Connection header received
0927 07:04:29.401412 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.401413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.401422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.401488 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.402200 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.402964 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404134 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.404746 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404812 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.407904 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.407906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.407906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.407913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.409710 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.410328 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.410329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.410329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.410389 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.411171 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.411173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.411173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.411318 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.411383 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.411584 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.411682 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:04:29.412415 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.399985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.21.2.66:40949)
0927 07:04:29.400378 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.400446 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.401240 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.401504 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.401505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.401887 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.402093 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.402099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.403058 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.403059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.404046 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.404049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.404870 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.404934 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.405400 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.405410 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.407808 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.409816 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.409823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.409826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.410232 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.411013 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.411014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.411014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.411096 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.411430 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.411430 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.411475 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.411907 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.412130 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":66,"thread_start_us":98,"threads_started":1}
I20260927 07:04:29.412648 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.400016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49381 (local address 127.21.2.125:44679)
0927 07:04:29.400290 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.400292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.400552 (+ 260us) server_negotiation.cc:407] Connection header received
0927 07:04:29.401546 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.401547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.401559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.401607 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.402200 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.402957 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404465 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.404891 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404998 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.408226 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.408228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.408228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.408236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.410041 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.411539 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.411541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.411541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.411599 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.412130 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.412131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.412132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.412252 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.412326 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.412486 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.412562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260927 07:04:29.412657 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.400000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.21.2.66:49381)
0927 07:04:29.400498 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.400508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.401240 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.401651 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.401652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.401891 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.402093 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.402099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.403059 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.403059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.404340 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.404342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.405431 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.405432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.405523 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.405928 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.405933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.408136 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.410127 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.410127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.410128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.411453 (+ 1325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.411673 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.411674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.411674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.411882 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.412330 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.412331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.412373 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.412449 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.412549 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":59,"thread_start_us":101,"threads_started":1}
I20260927 07:04:29.412837 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.399985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:51421)
0927 07:04:29.400168 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.400209 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.401228 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.401468 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.401472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.401887 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.402091 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.402095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.402794 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.402795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.403633 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.403635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.404018 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.404019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.404176 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.404720 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.404730 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.408025 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.409928 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.409929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.409930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.410268 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.410520 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.410521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.410521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.410645 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.411034 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.411040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.411228 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.411849 (+ 621us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.412679 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":584,"spinlock_wait_cycles":7424,"thread_start_us":109,"threads_started":1}
I20260927 07:04:29.413250 28711 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:29.413344 28711 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.413527 28711 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:29.413964 27609 ts_manager.cc:194] Re-registered known tserver with Master: 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287)
I20260927 07:04:29.414325 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:51421
I20260927 07:04:29.415598 28710 heartbeater.cc:347] Connected to a master server at 127.21.2.124:35019
I20260927 07:04:29.415697 28710 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.415781 28710 heartbeater.cc:511] Master 127.21.2.124:35019 requested a full tablet report, sending...
I20260927 07:04:29.416055 28394 ts_manager.cc:194] Registered new tserver with Master: 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287)
I20260927 07:04:29.416355 28712 heartbeater.cc:347] Connected to a master server at 127.21.2.125:44679
I20260927 07:04:29.416411 28712 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.416410 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:36053
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:45915
--webserver_port=42487
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:04:29.416471 28712 heartbeater.cc:511] Master 127.21.2.125:44679 requested a full tablet report, sending...
I20260927 07:04:29.416734 28208 ts_manager.cc:194] Registered new tserver with Master: 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287)
W20260927 07:04:29.497938 28717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:04:29.498135 28717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:04:29.498158 28717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:04:29.499461 28717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:04:29.499502 28717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:04:29.499537 28717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:04:29.501125 28717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=42487
--tserver_master_addrs=127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:04:29.501374 28717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:04:29.501595 28717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:04:29.506472 28723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.506614 28717 server_base.cc:1034] running on GCE node
W20260927 07:04:29.506438 28722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:04:29.506428 28725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:04:29.506886 28717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:04:29.507105 28717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:04:29.508226 28717 hybrid_clock.cc:648] HybridClock initialized: now 1790492669508212 us; error 30 us; skew 500 ppm
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:04:29.510880 28717 init.cc:377] Logged in from keytab as oryx/127.21.2.67@KRBTEST.COM (short username oryx)
I20260927 07:04:29.511343 28717 webserver.cc:460] Webserver started at http://127.21.2.67:42487/ using document root <none> and password file <none>
I20260927 07:04:29.511631 28717 fs_manager.cc:359] Metadata directory not provided
I20260927 07:04:29.511688 28717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:04:29.513543 28717 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:04:29.514298 28732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.514569 28717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:04:29.514671 28717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "8eb57ed582564f78bada13ceec7368bd"
format_stamp: "Formatted at 2026-09-27 07:04:28 on dist-test-slave-ltqz"
server_key: "4f5a9ae5b172fc2cf7a067ab5b17f40c"
server_key_iv: "d28df4e3e13f4174f762b431628ac954"
server_key_version: "encryptionkey@0"
I20260927 07:04:29.515087 28717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:04:29.522950 28717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:04:29.525128 28717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:04:29.525234 28717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:04:29.525394 28717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:04:29.525635 28717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:04:29.525665 28717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.525686 28717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:04:29.525699 28717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:04:29.531424 28717 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:36053
I20260927 07:04:29.531471 28845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:36053 every 8 connection(s)
I20260927 07:04:29.531801 28717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.125@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
Sep 27 07:04:29 dist-test-slave-ltqz krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492669, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.124@KRBTEST.COM
I20260927 07:04:29.541337 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 28717
I20260927 07:04:29.543622 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35157 (local address 127.21.2.125:44679)
0927 07:04:29.532828 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.532831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.532843 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:04:29.533985 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.533987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.534000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.534049 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.535022 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535720 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.536949 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537142 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537216 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.539696 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.539697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.539698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.539706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.542080 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542704 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.542705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.542706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.542758 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542962 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.542963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.542964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.543104 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.543143 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.543432 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.543527 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:29.543637 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:50469 (local address 127.21.2.126:45915)
0927 07:04:29.532624 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.532626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.533071 (+ 445us) server_negotiation.cc:407] Connection header received
0927 07:04:29.534014 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.534016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.534028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.534058 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.535045 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535801 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537231 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537461 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.540158 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.540160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.540160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.540169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.541879 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542470 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.542471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.542471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.542526 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542847 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.542848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.542848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.542988 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.543033 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.543435 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.543527 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:04:29.543826 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35597 (local address 127.21.2.124:35019)
0927 07:04:29.532844 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.532846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.532855 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:29.534022 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.534023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.534036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.534070 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.534993 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535801 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537196 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537476 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537542 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.540653 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.540654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.540655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.540662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.542231 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542737 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.542738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.542738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.542800 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.543057 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.543058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.543059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.543196 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.543244 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.543625 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.543728 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:04:29.544193 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:50469)
0927 07:04:29.532982 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.532992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.533884 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.534146 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.534152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.534673 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.534884 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.534889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535909 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.535910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.537107 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.537108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537534 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.537602 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.538111 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.538116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.540030 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.541975 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.541982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.541984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542374 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.542595 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.542596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542765 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.543080 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.543087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.543247 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.543753 (+ 506us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.543948 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":38,"thread_start_us":80,"threads_started":1}
I20260927 07:04:29.544193 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:44679 (local address 127.21.2.67:35157)
0927 07:04:29.532791 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.532800 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.533873 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.534137 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.534142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.534689 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.534885 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.534891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535921 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.535922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.536845 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.536847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537278 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.537425 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.537942 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.537950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.539602 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.542159 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.542160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542628 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.542836 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.542837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542892 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.543185 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.543185 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.543247 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.543773 (+ 526us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.544002 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":169,"thread_start_us":90,"threads_started":1}
I20260927 07:04:29.544608 28848 heartbeater.cc:347] Connected to a master server at 127.21.2.125:44679
I20260927 07:04:29.544620 28847 heartbeater.cc:347] Connected to a master server at 127.21.2.126:45915
I20260927 07:04:29.544704 28848 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.544704 28847 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.544898 28848 heartbeater.cc:511] Master 127.21.2.125:44679 requested a full tablet report, sending...
I20260927 07:04:29.544898 28847 heartbeater.cc:511] Master 127.21.2.126:45915 requested a full tablet report, sending...
I20260927 07:04:29.545285 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.532481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:35019 (local address 127.21.2.67:35597)
0927 07:04:29.532689 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.532730 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.533874 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.534148 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.534148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.534686 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:04:29.534885 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.534891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.535909 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.535910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.537105 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.537108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.537596 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.537597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.537663 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.538288 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:04:29.538293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:04:29.540567 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:04:29.542375 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.542376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542646 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.542883 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:04:29.542883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:04:29.542884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:04:29.542967 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:04:29.543296 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:04:29.543297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:04:29.543359 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.544949 (+ 1590us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.545091 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":63,"thread_start_us":101,"threads_started":1}
I20260927 07:04:29.545337 27609 ts_manager.cc:194] Re-registered known tserver with Master: 8eb57ed582564f78bada13ceec7368bd (127.21.2.67:36053)
I20260927 07:04:29.545395 28208 ts_manager.cc:194] Registered new tserver with Master: 8eb57ed582564f78bada13ceec7368bd (127.21.2.67:36053)
I20260927 07:04:29.545804 27609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:50469
I20260927 07:04:29.546394 28846 heartbeater.cc:347] Connected to a master server at 127.21.2.124:35019
I20260927 07:04:29.546451 28846 heartbeater.cc:464] Registering TS with master...
I20260927 07:04:29.546777 28846 heartbeater.cc:511] Master 127.21.2.124:35019 requested a full tablet report, sending...
I20260927 07:04:29.547619 28394 ts_manager.cc:194] Registered new tserver with Master: 8eb57ed582564f78bada13ceec7368bd (127.21.2.67:36053)
I20260927 07:04:29.553470 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.543087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.21.2.124:35019)
0927 07:04:29.543910 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.543912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.545389 (+ 1477us) server_negotiation.cc:407] Connection header received
0927 07:04:29.545518 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.545520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.545558 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.545598 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.546442 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.547185 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.548155 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.548458 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.548521 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.550211 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.550213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.550213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.550220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.552178 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.552588 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.552590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.552590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.552649 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.552841 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.552842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.552842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.552967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.553082 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.553285 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.553370 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":61,"threads_started":1}
I20260927 07:04:29.554468 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.21.2.125:44679)
0927 07:04:29.543704 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.543706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.544064 (+ 358us) server_negotiation.cc:407] Connection header received
0927 07:04:29.544173 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.544174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.544186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.544214 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.546485 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.547185 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.548208 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.548442 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.548509 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.550643 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.550644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.550644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.550651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.552903 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.553381 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.553383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.553383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.553476 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.553797 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.553798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.553799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.553965 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.554084 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.554308 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.554389 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":59,"threads_started":1}
I20260927 07:04:29.555467 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.543606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.21.2.126:45915)
0927 07:04:29.543707 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.543709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.548351 (+ 4642us) server_negotiation.cc:407] Connection header received
0927 07:04:29.548394 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.548395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.548407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.548462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.549120 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.549624 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.550326 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.550584 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.550644 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.552028 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.552029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.552030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.552038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.554054 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.554648 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.554649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.554649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.554715 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.554914 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.554916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.554916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.555055 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.555157 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.555263 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.555351 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":93,"threads_started":1}
I20260927 07:04:29.557132 27609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:04:29.558732 27609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:04:29.566010 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34707 (local address 127.0.0.1:49980)
0927 07:04:29.562632 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.562642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.562685 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.562921 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.562922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.562925 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.562984 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.562985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564223 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.564224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.564972 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.564973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.565759 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.565760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.565824 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.565828 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.565887 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.565934 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260927 07:04:29.566030 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.21.2.65:34707)
0927 07:04:29.562776 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.562780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.562789 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:29.562828 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.562830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.562841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.562925 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.563047 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564139 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.565058 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.565638 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.565755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.565834 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.565902 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.565942 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260927 07:04:29.566789 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.21.2.66:46287)
0927 07:04:29.563187 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.563191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.563201 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:04:29.563242 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.563244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.563259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.563335 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.563484 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564146 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.565475 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.566486 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.566548 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.566596 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.566703 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.566731 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":79,"threads_started":1}
I20260927 07:04:29.566804 28167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46287 (local address 127.0.0.1:50020)
0927 07:04:29.562409 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.562419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.562457 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.563341 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.563342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.563343 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.563413 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.563414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564268 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.564269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.565379 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.565381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.566565 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.566566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.566628 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.566632 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.566691 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.566734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":33}
I20260927 07:04:29.566751 28507 tablet_service.cc:1467] Processing CreateTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f (DEFAULT_TABLE table=test-table [id=4b53a5210fa941a489bde6c5da22924e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:29.567091 28507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d4eb14c84d4888a8af5bf16e57a90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:29.567816 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:36053 (local address 127.0.0.1:42104)
0927 07:04:29.562334 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.562345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.562398 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.562731 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.562733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.562736 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.562790 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.562792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564138 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.564140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.565379 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.565382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.567554 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.567555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.567619 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.567622 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.567675 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.567714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":70}
I20260927 07:04:29.568048 28644 tablet_service.cc:1467] Processing CreateTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f (DEFAULT_TABLE table=test-table [id=4b53a5210fa941a489bde6c5da22924e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:29.568229 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.562395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.21.2.67:36053)
0927 07:04:29.562546 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.562551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.562560 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:29.562604 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.562606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.562622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.562870 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.562937 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.564039 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.565488 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.566232 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.566319 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.566392 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.568077 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.568128 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260927 07:04:29.568375 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d4eb14c84d4888a8af5bf16e57a90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:29.568718 28780 tablet_service.cc:1467] Processing CreateTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f (DEFAULT_TABLE table=test-table [id=4b53a5210fa941a489bde6c5da22924e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:04:29.569002 28780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d4eb14c84d4888a8af5bf16e57a90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:04:29.570557 28871 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Bootstrap starting.
I20260927 07:04:29.571317 28871 tablet_bootstrap.cc:654] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:29.571691 28871 log.cc:824] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:29.572225 28873 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Bootstrap starting.
I20260927 07:04:29.572638 28871 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: No bootstrap required, opened a new log
I20260927 07:04:29.572707 28871 ts_tablet_manager.cc:1397] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:04:29.572845 28873 tablet_bootstrap.cc:654] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:29.573163 28873 log.cc:824] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:29.573489 28872 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Bootstrap starting.
I20260927 07:04:29.573848 28873 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: No bootstrap required, opened a new log
I20260927 07:04:29.573925 28873 ts_tablet_manager.cc:1397] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:29.574101 28872 tablet_bootstrap.cc:654] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Neither blocks nor log segments found. Creating new log.
I20260927 07:04:29.574410 28872 log.cc:824] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:04:29.575208 28872 tablet_bootstrap.cc:492] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: No bootstrap required, opened a new log
I20260927 07:04:29.575278 28872 ts_tablet_manager.cc:1397] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:04:29.575560 28871 raft_consensus.cc:348] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.575736 28871 raft_consensus.cc:374] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:29.575771 28871 raft_consensus.cc:729] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435025c5658f4879931d3d90a806516e, State: Initialized, Role: FOLLOWER
I20260927 07:04:29.575759 28873 raft_consensus.cc:348] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.575889 28871 consensus_queue.cc:260] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.575922 28873 raft_consensus.cc:374] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:29.575963 28873 raft_consensus.cc:729] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb57ed582564f78bada13ceec7368bd, State: Initialized, Role: FOLLOWER
I20260927 07:04:29.576071 28873 consensus_queue.cc:260] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.576128 28871 ts_tablet_manager.cc:1428] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 07:04:29.576344 28873 ts_tablet_manager.cc:1428] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:04:29.576447 28847 heartbeater.cc:503] Master 127.21.2.126:45915 was elected leader, sending a full tablet report...
I20260927 07:04:29.576525 28574 heartbeater.cc:503] Master 127.21.2.126:45915 was elected leader, sending a full tablet report...
I20260927 07:04:29.577869 28872 raft_consensus.cc:348] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.578033 28872 raft_consensus.cc:374] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:04:29.578082 28872 raft_consensus.cc:729] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404f3a2773524e019ca10a01fc0e277d, State: Initialized, Role: FOLLOWER
I20260927 07:04:29.578181 28872 consensus_queue.cc:260] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.578433 28872 ts_tablet_manager.cc:1428] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 07:04:29.578646 28711 heartbeater.cc:503] Master 127.21.2.126:45915 was elected leader, sending a full tablet report...
I20260927 07:04:29.594326 28878 raft_consensus.cc:482] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:04:29.594432 28878 raft_consensus.cc:504] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.594813 28878 leader_election.cc:288] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287), 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
I20260927 07:04:29.597638 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.595025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46287 (local address 127.21.2.67:60105)
0927 07:04:29.595083 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.595097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.595170 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.595311 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.595312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.595314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.595420 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.595423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.596075 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.596076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.596851 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.596853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.597366 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.597368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.597465 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.597469 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.597516 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.597546 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:04:29.597651 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.595123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:60105 (local address 127.21.2.66:46287)
0927 07:04:29.595173 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.595176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.595208 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:04:29.595238 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.595238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.595250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.595302 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.595493 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.596003 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.596951 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.597295 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.597345 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.597353 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.597535 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.597582 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:04:29.597723 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.595238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:58709 (local address 127.21.2.65:34707)
0927 07:04:29.595280 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.595282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.595291 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:04:29.595315 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.595316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.595327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.595435 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:04:29.595605 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.596070 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.596965 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.597365 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.597432 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.597441 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.597605 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.597649 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:04:29.597764 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.595135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34707 (local address 127.21.2.67:58709)
0927 07:04:29.595178 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:29.595188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:04:29.595224 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:04:29.595446 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:04:29.595446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:04:29.595448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:04:29.595533 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.595535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.596144 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.596145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.596852 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:04:29.596853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.597446 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.597447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:04:29.597525 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.597529 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:04:29.597586 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 07:04:29.597621 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260927 07:04:29.598033 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d4eb14c84d4888a8af5bf16e57a90f" candidate_uuid: "8eb57ed582564f78bada13ceec7368bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435025c5658f4879931d3d90a806516e" is_pre_election: true
I20260927 07:04:29.598035 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d4eb14c84d4888a8af5bf16e57a90f" candidate_uuid: "8eb57ed582564f78bada13ceec7368bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f3a2773524e019ca10a01fc0e277d" is_pre_election: true
I20260927 07:04:29.598183 28664 raft_consensus.cc:2454] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb57ed582564f78bada13ceec7368bd in term 0.
I20260927 07:04:29.598183 28527 raft_consensus.cc:2454] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb57ed582564f78bada13ceec7368bd in term 0.
I20260927 07:04:29.598421 28735 leader_election.cc:302] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435025c5658f4879931d3d90a806516e, 8eb57ed582564f78bada13ceec7368bd; no voters:
I20260927 07:04:29.598579 28878 raft_consensus.cc:2781] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:04:29.598641 28878 raft_consensus.cc:482] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:04:29.598663 28878 raft_consensus.cc:3037] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:29.599478 28878 raft_consensus.cc:504] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.599686 28878 leader_election.cc:288] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [CANDIDATE]: Term 1 election: Requested vote from peers 404f3a2773524e019ca10a01fc0e277d (127.21.2.66:46287), 435025c5658f4879931d3d90a806516e (127.21.2.65:34707)
I20260927 07:04:29.599754 28664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d4eb14c84d4888a8af5bf16e57a90f" candidate_uuid: "8eb57ed582564f78bada13ceec7368bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f3a2773524e019ca10a01fc0e277d"
I20260927 07:04:29.599872 28664 raft_consensus.cc:3037] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:29.599884 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d4eb14c84d4888a8af5bf16e57a90f" candidate_uuid: "8eb57ed582564f78bada13ceec7368bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435025c5658f4879931d3d90a806516e"
I20260927 07:04:29.599983 28527 raft_consensus.cc:3037] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:04:29.600672 28664 raft_consensus.cc:2454] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb57ed582564f78bada13ceec7368bd in term 1.
I20260927 07:04:29.600704 28527 raft_consensus.cc:2454] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb57ed582564f78bada13ceec7368bd in term 1.
I20260927 07:04:29.600863 28735 leader_election.cc:302] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435025c5658f4879931d3d90a806516e, 8eb57ed582564f78bada13ceec7368bd; no voters:
I20260927 07:04:29.600979 28878 raft_consensus.cc:2781] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:04:29.601146 28878 raft_consensus.cc:686] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 1 LEADER]: Becoming Leader. State: Replica: 8eb57ed582564f78bada13ceec7368bd, State: Running, Role: LEADER
I20260927 07:04:29.601379 28878 consensus_queue.cc:237] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } }
I20260927 07:04:29.602195 27608 catalog_manager.cc:5495] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb57ed582564f78bada13ceec7368bd (127.21.2.67). New cstate: current_term: 1 leader_uuid: "8eb57ed582564f78bada13ceec7368bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb57ed582564f78bada13ceec7368bd" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 36053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260927 07:04:29.612401 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.610221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.21.2.67:36053)
0927 07:04:29.610269 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.610271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.610283 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:04:29.610319 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.610320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.610344 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.610406 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:04:29.610507 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.610955 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.611753 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.611955 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.612015 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.612139 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:04:29.612234 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:04:29.612298 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.612327 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.612347 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:04:29.614225 28527 raft_consensus.cc:1261] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 1 FOLLOWER]: Refusing update from remote peer 8eb57ed582564f78bada13ceec7368bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:29.614257 28664 raft_consensus.cc:1261] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 1 FOLLOWER]: Refusing update from remote peer 8eb57ed582564f78bada13ceec7368bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:04:29.614593 28878 consensus_queue.cc:1035] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "435025c5658f4879931d3d90a806516e" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:29.614713 28880 consensus_queue.cc:1035] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "404f3a2773524e019ca10a01fc0e277d" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:04:29.617399 28883 mvcc.cc:205] Tried to move back new op lower bound from 7333857974736969728 to 7333857974688043008. Current Snapshot: MvccSnapshot[applied={T|T < 7333857974736969728}]
I20260927 07:04:29.617971 28885 mvcc.cc:205] Tried to move back new op lower bound from 7333857974736969728 to 7333857974688043008. Current Snapshot: MvccSnapshot[applied={T|T < 7333857974736969728}]
I20260927 07:04:29.618500 28884 mvcc.cc:205] Tried to move back new op lower bound from 7333857974736969728 to 7333857974688043008. Current Snapshot: MvccSnapshot[applied={T|T < 7333857974736969728}]
I20260927 07:04:29.620457 27609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:29.620532 27609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:04:29.621893 27609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 287e9eeb51094b37844e0a0ad282e4ec: Sending DeleteTablet for 3 replicas of tablet 36d4eb14c84d4888a8af5bf16e57a90f
I20260927 07:04:29.622233 28780 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:29 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:42104
I20260927 07:04:29.622466 28893 tablet_replica.cc:330] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: stopping tablet replica
I20260927 07:04:29.622575 28893 raft_consensus.cc:2229] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:04:29.622670 28893 raft_consensus.cc:2258] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:29.623077 28893 ts_tablet_manager.cc:1905] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:29.623067 28644 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:29 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50020
I20260927 07:04:29.623343 28895 tablet_replica.cc:330] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: stopping tablet replica
I20260927 07:04:29.623354 28507 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d4eb14c84d4888a8af5bf16e57a90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:04:29 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:49980
I20260927 07:04:29.623540 28895 raft_consensus.cc:2229] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:29.623682 28895 raft_consensus.cc:2258] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:29.624055 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 27577
I20260927 07:04:29.625516 28893 ts_tablet_manager.cc:1918] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:29.625530 28895 ts_tablet_manager.cc:1905] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:04:29.625602 28893 log.cc:1196] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/36d4eb14c84d4888a8af5bf16e57a90f
I20260927 07:04:29.625864 28893 ts_tablet_manager.cc:1939] T 36d4eb14c84d4888a8af5bf16e57a90f P 8eb57ed582564f78bada13ceec7368bd: Deleting consensus metadata
I20260927 07:04:29.627291 28895 ts_tablet_manager.cc:1918] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:29.627439 28895 log.cc:1196] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/36d4eb14c84d4888a8af5bf16e57a90f
I20260927 07:04:29.627764 28895 ts_tablet_manager.cc:1939] T 36d4eb14c84d4888a8af5bf16e57a90f P 404f3a2773524e019ca10a01fc0e277d: Deleting consensus metadata
I20260927 07:04:29.628309 28896 tablet_replica.cc:330] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: stopping tablet replica
I20260927 07:04:29.628427 28896 raft_consensus.cc:2229] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:04:29.628547 28896 raft_consensus.cc:2258] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:04:29.629595 28896 ts_tablet_manager.cc:1905] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 07:04:29.629937 28600 connection.cc:535] server connection from 127.0.0.1:50020 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 07:04:29.630143 28733 connection.cc:535] server connection from 127.0.0.1:42104 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:04:29.631424 28896 ts_tablet_manager.cc:1918] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:04:29.631567 28896 log.cc:1196] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/36d4eb14c84d4888a8af5bf16e57a90f
I20260927 07:04:29.631887 28896 ts_tablet_manager.cc:1939] T 36d4eb14c84d4888a8af5bf16e57a90f P 435025c5658f4879931d3d90a806516e: Deleting consensus metadata
W20260927 07:04:29.632288 28462 connection.cc:441] server connection from 127.0.0.1:49980 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49980 (request call id 1) could send its response
I20260927 07:04:29.635643 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.623686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.21.2.124:35019)
0927 07:04:29.623730 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.623732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.623740 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:04:29.623760 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.623761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.623771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.623829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.628236 (+ 4407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.629016 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.630233 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.630549 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.630593 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.632530 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.632532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.632532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.632540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.634226 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.634621 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.634623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.634623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.634677 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.634863 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.634864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.634864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.634996 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.635081 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.635467 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.635544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:04:29.640777 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.631982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.21.2.125:44679)
0927 07:04:29.632030 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.632032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.633041 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 07:04:29.633153 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.633154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.633167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.633197 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.634081 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.634772 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.635748 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.636038 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.636079 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.637888 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.637889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.637890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.637897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.639613 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.639942 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.639943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.639944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.640001 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.640199 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.640201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.640201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.640361 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.640461 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.640558 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.640655 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:04:29.640978 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:29.632343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.21.2.124:35019)
0927 07:04:29.632382 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:04:29.632384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:04:29.632772 (+ 388us) server_negotiation.cc:407] Connection header received
0927 07:04:29.632870 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:04:29.632871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:04:29.632883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:04:29.632921 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:04:29.633784 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.634573 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.635741 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:04:29.636075 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:04:29.636113 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:04:29.637909 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:04:29.637911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:04:29.637911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:04:29.637918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:04:29.639621 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.640134 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.640135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.640136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.640194 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:04:29.640363 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:04:29.640364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:04:29.640364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:04:29.640499 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:04:29.640585 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:04:29.640776 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 07:04:29.640862 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:04:30.618991 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:30.618684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35773)
0927 07:04:30.618843 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:30.618934 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
W20260927 07:04:30.619282 28574 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:30.619582 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:30.619146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:32937)
0927 07:04:30.619461 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:30.619528 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":63,"threads_started":1}
W20260927 07:04:30.619760 28847 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:30.619908 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:30.619687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:42605)
0927 07:04:30.619802 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:30.619865 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
W20260927 07:04:30.620067 28711 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:45915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:31.123129 28907 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:31.123250 28907 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:31.123571 28907 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:31.123863 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:31.123701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39242)
0927 07:04:31.123793 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:31.123828 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":46,"threads_started":1}
W20260927 07:04:31.124009 28193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:31.124403 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:31.124314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39246)
0927 07:04:31.124360 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:31.124385 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:04:31.124493 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:31.124588 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:31.124692 28907 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:31.619988 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:31.619695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:46057)
0927 07:04:31.619859 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:31.619930 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260927 07:04:31.620316 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:31.620068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:39355)
0927 07:04:31.620227 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:31.620267 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260927 07:04:31.620714 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:31.620386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:36621)
0927 07:04:31.620616 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:31.620651 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":50,"threads_started":1}
I20260927 07:04:32.620728 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:32.620436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:51339)
0927 07:04:32.620588 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:32.620670 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260927 07:04:32.621220 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:32.620845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:48289)
0927 07:04:32.621140 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:32.621175 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":63,"threads_started":1}
I20260927 07:04:32.621426 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:32.621111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:44623)
0927 07:04:32.621339 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:32.621376 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":51,"threads_started":1}
I20260927 07:04:32.785802 28916 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:32.785895 28916 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:32.786049 28916 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:32.786387 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:32.786192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39454)
0927 07:04:32.786302 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:32.786346 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 07:04:32.786837 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:32.786744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39460)
0927 07:04:32.786779 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:32.786816 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 07:04:32.786924 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:32.786970 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:32.787055 28916 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:33.621678 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:33.621403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:41325)
0927 07:04:33.621566 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:33.621617 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260927 07:04:33.621737 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:33.621506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:41201)
0927 07:04:33.621648 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:33.621687 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":47,"threads_started":1}
I20260927 07:04:33.622167 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:33.621947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:37719)
0927 07:04:33.622070 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:33.622123 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 07:04:34.622489 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:34.622261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35923)
0927 07:04:34.622393 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:34.622430 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260927 07:04:34.622490 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:34.622168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:51357)
0927 07:04:34.622383 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:34.622430 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260927 07:04:34.622854 28924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:34.622566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:37365)
0927 07:04:34.622767 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:34.622802 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":55,"threads_started":1}
I20260927 07:04:35.119495 28925 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:35.119596 28925 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:35.119740 28925 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:35.120056 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:35.119881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39730)
0927 07:04:35.119979 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:35.120020 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260927 07:04:35.120443 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:35.120361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:39738)
0927 07:04:35.120404 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:35.120428 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:04:35.120517 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:35.120591 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:35.120661 28925 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:35.623559 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:35.623263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:53361)
0927 07:04:35.623463 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:35.623505 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
I20260927 07:04:35.623556 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:35.622959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:36793)
0927 07:04:35.623450 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:35.623497 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":77,"threads_started":1}
I20260927 07:04:35.623656 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:35.623239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35023)
0927 07:04:35.623581 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:35.623616 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":52,"threads_started":1}
I20260927 07:04:36.624392 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:36.624090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:60435)
0927 07:04:36.624286 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:36.624330 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260927 07:04:36.624559 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:36.624299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:59559)
0927 07:04:36.624459 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:36.624504 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260927 07:04:36.624804 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:36.624623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:47105)
0927 07:04:36.624733 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:36.624771 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260927 07:04:37.625381 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:37.625157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:44083)
0927 07:04:37.625292 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:37.625328 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":55,"threads_started":1}
I20260927 07:04:37.625380 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:37.625000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:52487)
0927 07:04:37.625288 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:37.625328 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":58,"threads_started":1}
I20260927 07:04:37.625416 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:37.625177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:39449)
0927 07:04:37.625331 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:37.625372 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":50,"threads_started":1}
I20260927 07:04:38.626127 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:38.625811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:50561)
0927 07:04:38.626018 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:38.626061 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260927 07:04:38.626180 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:38.625973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:47957)
0927 07:04:38.626092 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:38.626130 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260927 07:04:38.626225 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:38.626013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:57265)
0927 07:04:38.626150 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:38.626185 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
I20260927 07:04:39.288780 28940 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:39.288908 28940 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:39.289100 28940 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:39.289451 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:39.289270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:60214)
0927 07:04:39.289377 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:39.289413 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
W20260927 07:04:39.289535 28193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260927 07:04:39.289913 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:39.289798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:60228)
0927 07:04:39.289860 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:39.289886 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 07:04:39.289994 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:39.290066 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:39.290158 28940 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:39.627043 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:39.626802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:42495)
0927 07:04:39.626940 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:39.626983 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 07:04:39.627164 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:39.626755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:50163)
0927 07:04:39.627064 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:39.627096 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":92,"threads_started":1}
I20260927 07:04:39.627197 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:39.627000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:51773)
0927 07:04:39.627103 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:39.627134 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260927 07:04:40.627836 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:40.627610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:37595)
0927 07:04:40.627738 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:40.627784 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260927 07:04:40.627836 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:40.627610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:45011)
0927 07:04:40.627759 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:40.627787 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260927 07:04:40.628065 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:40.627611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:57201)
0927 07:04:40.627885 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:40.628011 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":71,"threads_started":1}
I20260927 07:04:41.628636 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:41.628396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:40889)
0927 07:04:41.628545 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:41.628583 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260927 07:04:41.628636 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:41.628396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:39079)
0927 07:04:41.628542 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:41.628583 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260927 07:04:41.628854 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:41.628455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:43389)
0927 07:04:41.628696 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:41.628800 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":44,"threads_started":1}
I20260927 07:04:42.629398 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:42.629006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:58767)
0927 07:04:42.629162 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:42.629332 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260927 07:04:42.629593 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:42.629407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:43007)
0927 07:04:42.629512 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:42.629550 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260927 07:04:42.629593 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:42.629169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:35619)
0927 07:04:42.629517 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:42.629550 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":69,"threads_started":1}
I20260927 07:04:43.630338 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:43.630110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:52221)
0927 07:04:43.630235 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:43.630279 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":48,"threads_started":1}
I20260927 07:04:43.630360 28955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:43.629984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:50543)
0927 07:04:43.630235 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:43.630279 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260927 07:04:43.630549 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:43.630244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:33455)
0927 07:04:43.630387 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:43.630423 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260927 07:04:44.631206 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:44.630832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:46209)
0927 07:04:44.631104 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:44.631148 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":68,"threads_started":1}
I20260927 07:04:44.631270 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:44.630855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:47303)
0927 07:04:44.631177 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:44.631229 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":44,"threads_started":1}
I20260927 07:04:44.631306 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:44.630998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:54993)
0927 07:04:44.631177 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:44.631251 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260927 07:04:44.854899 28961 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:44.855010 28961 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:44.855134 28961 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:44.855561 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:44.855337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:60534)
0927 07:04:44.855479 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:44.855523 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
W20260927 07:04:44.855648 28193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:44.856006 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:44.855912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:60544)
0927 07:04:44.855962 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:44.855988 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 07:04:44.856096 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:44.856150 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:44.856240 28961 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:45.632004 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:45.631774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:47401)
0927 07:04:45.631905 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:45.631946 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260927 07:04:45.632046 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:45.631669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:40841)
0927 07:04:45.631955 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:45.631995 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":66,"threads_started":1}
I20260927 07:04:45.632066 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:45.631886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:58813)
0927 07:04:45.631991 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:45.632027 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260927 07:04:46.632771 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:46.632554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:41127)
0927 07:04:46.632678 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:46.632718 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260927 07:04:46.632771 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:46.632514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35259)
0927 07:04:46.632678 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:46.632718 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":44,"threads_started":1}
I20260927 07:04:46.632923 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:46.632461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:33391)
0927 07:04:46.632825 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:46.632868 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":74,"threads_started":1}
I20260927 07:04:47.633623 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:47.633375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:58023)
0927 07:04:47.633526 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:47.633568 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
I20260927 07:04:47.633623 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:47.633193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:60503)
0927 07:04:47.633515 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:47.633561 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":69,"threads_started":1}
I20260927 07:04:47.633718 28971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:47.633375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:50939)
0927 07:04:47.633640 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:47.633675 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":58,"threads_started":1}
I20260927 07:04:48.302951 28973 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 287e9eeb51094b37844e0a0ad282e4ec)
I20260927 07:04:48.303073 28973 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "287e9eeb51094b37844e0a0ad282e4ec" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45915 } } peers { permanent_uuid: "b2e64e84611f4e77ac9ed15d727fba9c" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 44679 } attrs { promote: false } } peers { permanent_uuid: "543c9ad1aa334e70a6f831d4cbf094fa" member_type: NON_VOTER last_known_addr { host: "127.21.2.124" port: 35019 } attrs { promote: true } }
I20260927 07:04:48.303277 28973 leader_election.cc:288] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915)
I20260927 07:04:48.303683 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:48.303491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38884)
0927 07:04:48.303601 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:48.303644 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260927 07:04:48.304126 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:48.304006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.0.0.1:38896)
0927 07:04:48.304071 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:48.304103 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 07:04:48.304229 28193 leader_election.cc:334] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 287e9eeb51094b37844e0a0ad282e4ec (127.21.2.126:45915): Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
I20260927 07:04:48.304282 28193 leader_election.cc:302] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b2e64e84611f4e77ac9ed15d727fba9c; no voters: 287e9eeb51094b37844e0a0ad282e4ec
I20260927 07:04:48.304395 28973 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b2e64e84611f4e77ac9ed15d727fba9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:04:48.634459 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:48.634039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:39373)
0927 07:04:48.634256 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:48.634312 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260927 07:04:48.634537 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:48.634167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:59373)
0927 07:04:48.634438 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:48.634489 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":59,"threads_started":1}
I20260927 07:04:48.634564 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:48.634354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:39287)
0927 07:04:48.634497 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:48.634523 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":46,"threads_started":1}
I20260927 07:04:49.635445 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:49.635225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:57359)
0927 07:04:49.635321 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:49.635362 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260927 07:04:49.635468 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:49.635042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:38341)
0927 07:04:49.635260 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:49.635306 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
I20260927 07:04:49.635669 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:49.635058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:48145)
0927 07:04:49.635476 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:49.635513 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":56,"threads_started":1}
I20260927 07:04:50.636340 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:50.636156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:42991)
0927 07:04:50.636250 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:50.636287 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260927 07:04:50.636356 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:50.636028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:37363)
0927 07:04:50.636248 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:50.636289 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":63,"threads_started":1}
I20260927 07:04:50.636350 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:50.636019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:41029)
0927 07:04:50.636153 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:50.636313 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260927 07:04:51.637179 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:51.636884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:43441)
0927 07:04:51.637076 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:51.637123 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 07:04:51.637235 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:51.636822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:49199)
0927 07:04:51.637159 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:51.637194 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":67,"threads_started":1}
I20260927 07:04:51.637271 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:51.637075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:51917)
0927 07:04:51.637181 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:51.637219 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260927 07:04:52.638182 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:52.637706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:57931)
0927 07:04:52.638083 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:52.638123 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":75,"threads_started":1}
I20260927 07:04:52.638199 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:52.637971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:41059)
0927 07:04:52.638089 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:52.638129 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":48,"threads_started":1}
I20260927 07:04:52.638239 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:52.637703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:53197)
0927 07:04:52.638033 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:52.638195 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":74,"threads_started":1}
I20260927 07:04:53.638948 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:53.638594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:55371)
0927 07:04:53.638784 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:53.638825 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260927 07:04:53.638975 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:53.638588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35983)
0927 07:04:53.638904 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:53.638934 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":77,"threads_started":1}
I20260927 07:04:53.639302 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:53.638934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:38621)
0927 07:04:53.639218 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:53.639255 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":40,"threads_started":1}
I20260927 07:04:54.639784 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:54.639477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:48707)
0927 07:04:54.639671 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:54.639724 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260927 07:04:54.640061 28995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:54.639741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:45803)
0927 07:04:54.639972 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:54.640008 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":51,"threads_started":1}
I20260927 07:04:54.640184 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:54.639983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:40605)
0927 07:04:54.640108 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:54.640140 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260927 07:04:55.640625 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:55.640365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:51389)
0927 07:04:55.640514 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:55.640562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260927 07:04:55.641062 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:55.640750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:48555)
0927 07:04:55.640981 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:55.641020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":69,"threads_started":1}
I20260927 07:04:55.641103 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:55.640545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:57435)
0927 07:04:55.641011 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:55.641041 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":64,"threads_started":1}
I20260927 07:04:56.641673 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:56.641230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:58259)
0927 07:04:56.641523 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:56.641610 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":67,"threads_started":1}
I20260927 07:04:56.641950 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:56.641525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:41383)
0927 07:04:56.641815 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:56.641854 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":69,"threads_started":1}
I20260927 07:04:56.642009 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:56.641674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:43197)
0927 07:04:56.641933 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:56.641967 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":62,"threads_started":1}
I20260927 07:04:57.642642 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:57.642344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:56925)
0927 07:04:57.642546 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:57.642588 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":66,"threads_started":1}
I20260927 07:04:57.642663 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:57.642374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:53725)
0927 07:04:57.642555 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:57.642588 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
I20260927 07:04:57.642828 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:57.642295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:51911)
0927 07:04:57.642733 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:57.642774 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":103,"threads_started":1}
I20260927 07:04:58.643508 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:58.643132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:36689)
0927 07:04:58.643391 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:58.643440 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
I20260927 07:04:58.643582 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:58.643411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:47471)
0927 07:04:58.643509 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:58.643536 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20260927 07:04:58.643582 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:58.643211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:41491)
0927 07:04:58.643497 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:58.643538 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":47,"threads_started":1}
I20260927 07:04:59.644383 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:59.644102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.66:34369)
0927 07:04:59.644276 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:59.644319 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260927 07:04:59.644469 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:59.644291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.65:35725)
0927 07:04:59.644397 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:59.644425 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":46,"threads_started":1}
I20260927 07:04:59.644722 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:04:59.644361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45915 (local address 127.21.2.67:48511)
0927 07:04:59.644485 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 07:04:59.644614 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.21.2.126:45915,127.21.2.125:44679,127.21.2.124:35019, num_attempts: 1)
I20260927 07:04:59.828325 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 28355
I20260927 07:04:59.833540 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 28581
I20260927 07:04:59.838442 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 28717
I20260927 07:04:59.843739 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 28073
I20260927 07:04:59.848371 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 28262
2026-09-27T07:04:59Z chronyd exiting
I20260927 07:04:59.865235 21513 test_util.cc:175] -----------------------------------------------
I20260927 07:04:59.865331 21513 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492641704908-21513-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32983 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:04:59 dist-test-slave-ltqz krb5kdc[29014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:04:59 dist-test-slave-ltqz krb5kdc[29014](info): set up 2 sockets
Sep 27 07:04:59 dist-test-slave-ltqz krb5kdc[29014](info): commencing operation
krb5kdc: starting...
W20260927 07:05:00.912289 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:05:00 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab.
Entry for principal oryx/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab.
I20260927 07:05:01.031136 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46793
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab
--principal=oryx/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:46793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:01.118271 29027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:01.118527 29027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:01.118564 29027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:01.119807 29027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:01.119874 29027 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:05:01.119897 29027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:01.119920 29027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:01.119967 29027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:01.119995 29027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:01.121526 29027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:46793
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.126.keytab
--principal=oryx/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:01.121816 29027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:01.122042 29027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:05:01.127559 29027 server_base.cc:1034] running on GCE node
W20260927 07:05:01.127664 29033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.127631 29035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.127645 29032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:01.128109 29027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:01.128366 29027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:01.129561 29027 hybrid_clock.cc:648] HybridClock initialized: now 1790492701129547 us; error 44 us; skew 500 ppm
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:01.135154 29027 init.cc:377] Logged in from keytab as oryx/127.21.2.126@KRBTEST.COM (short username oryx)
I20260927 07:05:01.135670 29027 webserver.cc:460] Webserver started at http://127.21.2.126:38351/ using document root <none> and password file <none>
I20260927 07:05:01.135929 29027 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:01.136003 29027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:01.136196 29027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:01.137269 29027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "39445fd5746f48b696ed4cd2b5a84a00"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "3fc1ba204a9c422414296a7affd18f7e"
server_key_iv: "0d0df7a733b7bdec290b86d48cb4619d"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.137641 29027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "39445fd5746f48b696ed4cd2b5a84a00"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "3fc1ba204a9c422414296a7affd18f7e"
server_key_iv: "0d0df7a733b7bdec290b86d48cb4619d"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.139011 29027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:01.139804 29042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.139981 29027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.140081 29027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "39445fd5746f48b696ed4cd2b5a84a00"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "3fc1ba204a9c422414296a7affd18f7e"
server_key_iv: "0d0df7a733b7bdec290b86d48cb4619d"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.140164 29027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:01.165750 29027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:01.168597 29027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:01.168740 29027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:01.175431 29027 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:46793
I20260927 07:05:01.175583 29104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:46793 every 8 connection(s)
I20260927 07:05:01.175856 29027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:01.176411 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29027
I20260927 07:05:01.176532 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:05:01.176683 29105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.176811 21513 external_mini_cluster.cc:1428] Setting key 15eb900a60b6680e3e034050d5fba554
I20260927 07:05:01.180151 29105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Bootstrap starting.
I20260927 07:05:01.180950 29105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.181324 29105 log.cc:824] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:01.182219 29105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: No bootstrap required, opened a new log
I20260927 07:05:01.184371 29105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } }
I20260927 07:05:01.184545 29105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.184614 29105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39445fd5746f48b696ed4cd2b5a84a00, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.184768 29105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } }
I20260927 07:05:01.184880 29105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:01.184937 29105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:01.184995 29105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:01.185523 29105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } }
I20260927 07:05:01.185649 29105 leader_election.cc:302] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39445fd5746f48b696ed4cd2b5a84a00; no voters:
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:01.185822 29105 leader_election.cc:288] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:05:01.185946 29110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:01.186098 29110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [term 1 LEADER]: Becoming Leader. State: Replica: 39445fd5746f48b696ed4cd2b5a84a00, State: Running, Role: LEADER
I20260927 07:05:01.186125 29105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:01.186223 29110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } }
I20260927 07:05:01.187175 29112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39445fd5746f48b696ed4cd2b5a84a00. Latest consensus state: current_term: 1 leader_uuid: "39445fd5746f48b696ed4cd2b5a84a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } } }
I20260927 07:05:01.187671 29112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:01.188333 29111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39445fd5746f48b696ed4cd2b5a84a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39445fd5746f48b696ed4cd2b5a84a00" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46793 } } }
W20260927 07:05:01.188482 29125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 07:05:01.188484 29111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00 [sys.catalog]: This master's current role is: LEADER
W20260927 07:05:01.188578 29125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:05:01.188674 29122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:05:01.188901 29122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:01.190908 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.177311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.21.2.126:46793)
0927 07:05:01.177505 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.177515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.177547 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:05:01.178083 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.178092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.178156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.178305 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.179326 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.180901 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.182116 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.182917 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.183063 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.186339 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.186344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.186344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.186357 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.188619 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.189051 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.189055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.189055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.189184 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.189428 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.189429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.189429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.189790 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.189955 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.190455 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.190650 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260927 07:05:01.192354 29122 catalog_manager.cc:1347] Generated new cluster ID: c806cf5c9de249ba946e5184ac96bb08
I20260927 07:05:01.192415 29122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:01.207726 29122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:05:01.208443 29122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:01.214190 29122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Generated new TSK 0
I20260927 07:05:01.214370 29122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
I20260927 07:05:01.243124 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:01.325028 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:01.325260 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:01.325300 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:01.326437 29131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:01.326504 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:01.326542 29131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:05:01.326601 29131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:01.328122 29131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:01.328377 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:01.328590 29131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:01.333448 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.333437 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.333576 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:01.333832 29131 server_base.cc:1034] running on GCE node
I20260927 07:05:01.334028 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:01.334311 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:01.335438 29131 hybrid_clock.cc:648] HybridClock initialized: now 1790492701335411 us; error 37 us; skew 500 ppm
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:01.337916 29131 init.cc:377] Logged in from keytab as oryx/127.21.2.65@KRBTEST.COM (short username oryx)
I20260927 07:05:01.338336 29131 webserver.cc:460] Webserver started at http://127.21.2.65:45411/ using document root <none> and password file <none>
I20260927 07:05:01.338559 29131 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:01.338630 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:01.338774 29131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:01.339883 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "8602f2ee9a2a44e18f5e1a574b03fe95"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "8aa452880fb09fd4d3a28d8f67d9b1d9"
server_key_iv: "344e2d5d9c436a6dc5c8835a68acef22"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.340250 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "8602f2ee9a2a44e18f5e1a574b03fe95"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "8aa452880fb09fd4d3a28d8f67d9b1d9"
server_key_iv: "344e2d5d9c436a6dc5c8835a68acef22"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.341586 29131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:05:01.342375 29146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.342597 29131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:01.342712 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "8602f2ee9a2a44e18f5e1a574b03fe95"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "8aa452880fb09fd4d3a28d8f67d9b1d9"
server_key_iv: "344e2d5d9c436a6dc5c8835a68acef22"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.342803 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:01.356094 29131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:01.357996 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:01.358186 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:01.358701 29131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:01.358769 29131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.358834 29131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:01.358865 29131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:01.366149 29131 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:34531
I20260927 07:05:01.366218 29261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:34531 every 8 connection(s)
I20260927 07:05:01.366544 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:05:01.368428 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29131
I20260927 07:05:01.368561 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:05:01.368779 21513 external_mini_cluster.cc:1428] Setting key a08e78a2259ab5fef988a7a54df39bf3
I20260927 07:05:01.369726 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.359266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:37799 (local address 127.21.2.126:46793)
0927 07:05:01.359326 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.359328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.359545 (+ 217us) server_negotiation.cc:407] Connection header received
0927 07:05:01.360092 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.360093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.360107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.360161 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.360814 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.361645 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.363385 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.363736 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.363815 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.366559 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.366560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.366561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.366569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.368183 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.368643 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.368644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.368645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.368716 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.369006 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.369007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.369007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.369152 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.369244 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.369547 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.369631 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:05:01.369928 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.359243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.65:37799)
0927 07:05:01.359416 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.359461 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.359977 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.360235 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.360238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.360531 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.360715 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.360719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.361766 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.361767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.363256 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.363261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.363860 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.363861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.364029 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.364528 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.364541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.366457 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.368290 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.368296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.368298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.368519 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.368798 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.368799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.368799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.368924 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.369235 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.369242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.369398 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.369525 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.369693 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260927 07:05:01.371022 29262 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46793
I20260927 07:05:01.371116 29262 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:01.371279 29262 heartbeater.cc:511] Master 127.21.2.126:46793 requested a full tablet report, sending...
I20260927 07:05:01.371848 29058 ts_manager.cc:194] Registered new tserver with Master: 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531)
I20260927 07:05:01.372475 29058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:37799
WARNING: no policy specified for oryx/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:05:01.378130 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.371406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:37753 (local address 127.21.2.126:46793)
0927 07:05:01.371452 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.371454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.371548 (+ 94us) server_negotiation.cc:407] Connection header received
0927 07:05:01.372696 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.372697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.372709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.372760 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.373198 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.373740 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.374384 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.374559 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.374619 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.375678 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.375679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.375680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.375685 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.377025 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.377427 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.377428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.377428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.377475 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.377701 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.377701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.377702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.377791 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.377840 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.378005 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.378060 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:05:01.378238 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.371322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.65:37753)
0927 07:05:01.371490 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.371501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.371556 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.372808 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.372809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.373022 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.373093 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.373096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.373836 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.373837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.374312 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.374313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.374657 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.374658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.374739 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.375133 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.375138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.375584 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.377117 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.377118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.377119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.377345 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.377547 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.377549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.377549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.377631 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.377857 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.377858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.377906 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.377978 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.378092 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
I20260927 07:05:01.394567 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:01.475693 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:01.475898 29271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:01.475921 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:01.477094 29271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:01.477135 29271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:01.477156 29271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:05:01.477202 29271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:01.478688 29271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:01.478935 29271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:01.479152 29271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:01.484073 29279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.484197 29276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.484299 29277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:01.484418 29271 server_base.cc:1034] running on GCE node
I20260927 07:05:01.484614 29271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:01.484815 29271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:01.485960 29271 hybrid_clock.cc:648] HybridClock initialized: now 1790492701485935 us; error 45 us; skew 500 ppm
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:01.488648 29271 init.cc:377] Logged in from keytab as oryx/127.21.2.66@KRBTEST.COM (short username oryx)
I20260927 07:05:01.489065 29271 webserver.cc:460] Webserver started at http://127.21.2.66:34537/ using document root <none> and password file <none>
I20260927 07:05:01.489267 29271 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:01.489307 29271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:01.489394 29271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:01.490388 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "71ef83a2beb84430bb771cc9fbb22224"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "2c3ad6b7cfa30075fe88e180fc85f1e7"
server_key_iv: "bb18cf4e2192915a8809942ba4e00c9c"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.490778 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "71ef83a2beb84430bb771cc9fbb22224"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "2c3ad6b7cfa30075fe88e180fc85f1e7"
server_key_iv: "bb18cf4e2192915a8809942ba4e00c9c"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.492261 29271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:05:01.493053 29286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.493292 29271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:05:01.493364 29271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "71ef83a2beb84430bb771cc9fbb22224"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "2c3ad6b7cfa30075fe88e180fc85f1e7"
server_key_iv: "bb18cf4e2192915a8809942ba4e00c9c"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.493458 29271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:01.527212 29271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:01.529781 29271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:01.529971 29271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:01.530489 29271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:01.530551 29271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.530603 29271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:01.530627 29271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:01.539119 29271 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:37303
I20260927 07:05:01.539156 29401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:37303 every 8 connection(s)
I20260927 07:05:01.539544 29271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:05:01.539853 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29271
I20260927 07:05:01.539971 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:05:01.540180 21513 external_mini_cluster.cc:1428] Setting key 0610fc9de5892a5fd4a2cbaad6afdbcd
I20260927 07:05:01.541857 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.531073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.66:48875)
0927 07:05:01.531268 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.531303 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.532051 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.532407 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.532412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.532791 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.533149 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.533157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.534159 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.534160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.535666 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.535669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.536077 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.536078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.536227 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.536779 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.536794 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.538587 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.540451 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.540459 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.540461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.540721 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.540948 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.540950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.540950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.541066 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.541373 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.541377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.541488 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.541570 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.541701 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260927 07:05:01.541863 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.531085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:48875 (local address 127.21.2.126:46793)
0927 07:05:01.531147 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.531149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.531410 (+ 261us) server_negotiation.cc:407] Connection header received
0927 07:05:01.532174 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.532176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.532190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.532279 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.533279 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.534047 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.535761 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.535976 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.536068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.538710 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.538712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.538712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.538720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.540326 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.540803 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.540804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.540805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.540873 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.541139 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.541140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.541140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.541302 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.541377 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.541622 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.541722 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:05:01.542433 29402 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46793
I20260927 07:05:01.542531 29402 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:01.542735 29402 heartbeater.cc:511] Master 127.21.2.126:46793 requested a full tablet report, sending...
I20260927 07:05:01.543205 29059 ts_manager.cc:194] Registered new tserver with Master: 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303)
I20260927 07:05:01.543763 29059 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:48875
WARNING: no policy specified for oryx/127.21.2.67@KRBTEST.COM; defaulting to no policy
I20260927 07:05:01.550491 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.543708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:54927 (local address 127.21.2.126:46793)
0927 07:05:01.543756 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.543759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.543925 (+ 166us) server_negotiation.cc:407] Connection header received
0927 07:05:01.544012 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.544013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.544025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.544088 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.544517 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.545068 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.546198 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.546451 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.546517 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.547775 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.547776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.547776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.547783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.549351 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.549768 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.549769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.549769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.549825 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.550043 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.550043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.550043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.550181 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.550237 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.550362 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.550425 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:05:01.550540 29409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.543637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.66:54927)
0927 07:05:01.543861 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.543872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.543936 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.544099 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.544100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.544330 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.544419 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.544421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.545181 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.545182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.546105 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.546109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.546543 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.546545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.546625 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.547175 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.547181 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.547645 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.549448 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.549449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.549450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.549700 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.549905 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.549907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.549907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.549975 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.550246 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.550247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.550283 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.550342 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.550424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
I20260927 07:05:01.566370 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:46793
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:01.648350 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:01.648558 29411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:01.648581 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:01.649724 29411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:01.649768 29411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:01.649788 29411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:05:01.649806 29411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:01.651386 29411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.2.126:46793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:01.651633 29411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:01.651887 29411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:01.656845 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:01.656841 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:01.656879 29411 server_base.cc:1034] running on GCE node
W20260927 07:05:01.656845 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:01.657256 29411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:01.657501 29411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:01.658706 29411 hybrid_clock.cc:648] HybridClock initialized: now 1790492701658692 us; error 25 us; skew 500 ppm
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:01.661266 29411 init.cc:377] Logged in from keytab as oryx/127.21.2.67@KRBTEST.COM (short username oryx)
I20260927 07:05:01.661746 29411 webserver.cc:460] Webserver started at http://127.21.2.67:34671/ using document root <none> and password file <none>
I20260927 07:05:01.661980 29411 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:01.662050 29411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:01.662204 29411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:01.663235 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "ce7eaf9ca10748cf9c53f686700c5188"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "e9755acc062ca70aafd7da2c53c84d5f"
server_key_iv: "9a2eebb05a0dcd693d465a3c17d3e5b5"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.663652 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "ce7eaf9ca10748cf9c53f686700c5188"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "e9755acc062ca70aafd7da2c53c84d5f"
server_key_iv: "9a2eebb05a0dcd693d465a3c17d3e5b5"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.665038 29411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:01.665854 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.666093 29411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:01.666208 29411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "ce7eaf9ca10748cf9c53f686700c5188"
format_stamp: "Formatted at 2026-09-27 07:05:01 on dist-test-slave-ltqz"
server_key: "e9755acc062ca70aafd7da2c53c84d5f"
server_key_iv: "9a2eebb05a0dcd693d465a3c17d3e5b5"
server_key_version: "encryptionkey@0"
I20260927 07:05:01.666299 29411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:01.678040 29411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:01.680303 29411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:01.680471 29411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:01.681135 29411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:01.681205 29411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.681247 29411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:01.681275 29411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:01.688872 29411 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:35303
I20260927 07:05:01.688961 29541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:35303 every 8 connection(s)
I20260927 07:05:01.689256 29411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:05:01.691289 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29411
I20260927 07:05:01.691404 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:05:01.691627 21513 external_mini_cluster.cc:1428] Setting key c35f70e62c068d2085fdf00679e26775
I20260927 07:05:01.692689 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.681638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:51549 (local address 127.21.2.126:46793)
0927 07:05:01.681719 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.681722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.681880 (+ 158us) server_negotiation.cc:407] Connection header received
0927 07:05:01.682452 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.682454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.682467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.682549 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.683282 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.684135 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.685209 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.685555 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.685640 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.688586 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.688587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.688588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.688595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.690519 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.691101 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.691102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.691102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.691191 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.691706 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.691707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.691708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.692037 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.692122 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.692501 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.692595 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:05:01.692951 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.681583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.67:51549)
0927 07:05:01.681735 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.681775 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.682344 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.682676 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.682685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.682973 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.683168 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.683172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.684266 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.684267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.685086 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.685088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.685654 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.685654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.685796 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.686360 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.686371 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.688450 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.690636 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.690641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.690644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.690936 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.691313 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.691314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.691314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.691593 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.692163 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.692170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.692343 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.692457 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.692654 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260927 07:05:01.693486 29542 heartbeater.cc:347] Connected to a master server at 127.21.2.126:46793
I20260927 07:05:01.693564 29542 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:01.693733 29542 heartbeater.cc:511] Master 127.21.2.126:46793 requested a full tablet report, sending...
I20260927 07:05:01.694151 29058 ts_manager.cc:194] Registered new tserver with Master: ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:01.694593 29058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:51549
I20260927 07:05:01.699991 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.694624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48023 (local address 127.21.2.126:46793)
0927 07:05:01.694667 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.694669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.694729 (+ 60us) server_negotiation.cc:407] Connection header received
0927 07:05:01.694799 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.694800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.694810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.694853 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.695201 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.695755 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.696540 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.696748 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.696834 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.697665 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.697667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.697667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.697674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.698767 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.699171 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.699172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.699173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.699232 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.699471 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.699472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.699473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.699584 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.699670 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.699826 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.699898 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:05:01.700006 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.694528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.21.2.67:48023)
0927 07:05:01.694669 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.694680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.694744 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.694866 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.694867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.695045 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:01.695102 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.695103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.695856 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.695856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.696446 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.696448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.696836 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.696837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.696900 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.697267 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:01.697272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:01.697569 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:01.698883 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.698884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.698885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.699060 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.699313 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:01.699314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:01.699314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:01.699419 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:01.699662 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:01.699663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:01.699707 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.699799 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.699900 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260927 07:05:01.703536 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:05:01.724902 29108 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:01.719329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.21.2.126:46793)
0927 07:05:01.719437 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.719439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.719448 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:01.719473 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.719473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.719482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.719534 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.720182 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.720665 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.721431 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.721618 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.721671 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.724804 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46970: BlockingRecv error: recv got EOF from 127.0.0.1:46970 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 07:05:01 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:01.732725 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.725695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.21.2.126:46793)
0927 07:05:01.725776 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.725777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.725849 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:05:01.725894 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.725895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.725903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.725970 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:01.726430 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.726906 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.727642 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.727823 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.727883 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.730096 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:01.730097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:01.730098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:01.730105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:01.731688 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.732030 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.732031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.732031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.732094 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:01.732327 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:01.732327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:01.732328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:01.732431 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:01.732507 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.732600 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.732658 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:05:01.734655 29058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:05:01.736420 29058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:01.743314 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.739029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.0.0.1:50798)
0927 07:05:01.739413 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.739422 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.739468 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.739733 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.739734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.739736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:01.739795 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.739797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.741126 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.741127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.742065 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.742067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.742916 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.743008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.743043 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.743144 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.743206 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":73,"threads_started":1}
I20260927 07:05:01.743315 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.738840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34531 (local address 127.0.0.1:34432)
0927 07:05:01.739172 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.739211 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.739297 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.739846 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.739847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.739849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:01.739892 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.739893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.740700 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.740701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.741948 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.741951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.743041 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.743042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.743108 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.743112 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.743163 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.743212 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260927 07:05:01.743538 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.739028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.21.2.67:35303)
0927 07:05:01.739507 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.739511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.739521 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:05:01.739562 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.739564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.739576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.739664 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:01.739836 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.740974 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742218 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.742964 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.743028 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.743094 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.743395 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.743438 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":89,"threads_started":1}
I20260927 07:05:01.743525 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.738949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.21.2.65:34531)
0927 07:05:01.739409 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.739414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.739426 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:05:01.739469 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.739471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.739487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.739854 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:01.739957 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.740606 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742037 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.742857 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742928 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.743002 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.743384 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.743426 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":88,"threads_started":1}
I20260927 07:05:01.743558 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.738840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35303 (local address 127.0.0.1:50074)
0927 07:05:01.739169 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.739213 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.739276 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.739670 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.739673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.739675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:01.739752 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.739754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.741081 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.741082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.742150 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.742152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.743139 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.743139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.743199 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.743202 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.743247 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.743282 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260927 07:05:01.743335 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.739159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.21.2.66:37303)
0927 07:05:01.739358 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.739383 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.739525 (+ 142us) server_negotiation.cc:407] Connection header received
0927 07:05:01.739578 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.739580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.739596 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.739666 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:01.739985 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.741029 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742158 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.742835 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.742914 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.742998 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.743160 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.743232 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260927 07:05:01.744311 29476 tablet_service.cc:1467] Processing CreateTablet for tablet bfca716b64eb468a9042ee83f384ae2c (DEFAULT_TABLE table=test-table [id=ff931ea0c038457eb7c8ab5ff691497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.744419 29196 tablet_service.cc:1467] Processing CreateTablet for tablet bfca716b64eb468a9042ee83f384ae2c (DEFAULT_TABLE table=test-table [id=ff931ea0c038457eb7c8ab5ff691497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.744699 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfca716b64eb468a9042ee83f384ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.744805 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfca716b64eb468a9042ee83f384ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.745020 29319 tablet_service.cc:1467] Processing CreateTablet for tablet bfca716b64eb468a9042ee83f384ae2c (DEFAULT_TABLE table=test-table [id=ff931ea0c038457eb7c8ab5ff691497e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.745357 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfca716b64eb468a9042ee83f384ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.748386 29570 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Bootstrap starting.
I20260927 07:05:01.748991 29571 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Bootstrap starting.
I20260927 07:05:01.749202 29570 tablet_bootstrap.cc:654] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.749485 29572 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Bootstrap starting.
I20260927 07:05:01.749640 29571 tablet_bootstrap.cc:654] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.749653 29570 log.cc:824] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:01.749982 29571 log.cc:824] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:01.750145 29572 tablet_bootstrap.cc:654] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.750480 29572 log.cc:824] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:01.750569 29570 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: No bootstrap required, opened a new log
I20260927 07:05:01.750643 29570 ts_tablet_manager.cc:1397] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:05:01.750785 29571 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: No bootstrap required, opened a new log
I20260927 07:05:01.750841 29571 ts_tablet_manager.cc:1397] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:05:01.752343 29570 raft_consensus.cc:348] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.752511 29570 raft_consensus.cc:374] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.752558 29570 raft_consensus.cc:729] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.752437 29571 raft_consensus.cc:348] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.752599 29571 raft_consensus.cc:374] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.752643 29571 raft_consensus.cc:729] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.752676 29570 consensus_queue.cc:260] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.752750 29571 consensus_queue.cc:260] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.752936 29570 ts_tablet_manager.cc:1428] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:05:01.752979 29542 heartbeater.cc:503] Master 127.21.2.126:46793 was elected leader, sending a full tablet report...
I20260927 07:05:01.753046 29571 ts_tablet_manager.cc:1428] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:05:01.753130 29262 heartbeater.cc:503] Master 127.21.2.126:46793 was elected leader, sending a full tablet report...
I20260927 07:05:01.753728 29572 tablet_bootstrap.cc:492] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: No bootstrap required, opened a new log
I20260927 07:05:01.753821 29572 ts_tablet_manager.cc:1397] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 07:05:01.755640 29572 raft_consensus.cc:348] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.755791 29572 raft_consensus.cc:374] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.755836 29572 raft_consensus.cc:729] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.755982 29572 consensus_queue.cc:260] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.756261 29572 ts_tablet_manager.cc:1428] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:05:01.756333 29402 heartbeater.cc:503] Master 127.21.2.126:46793 was elected leader, sending a full tablet report...
W20260927 07:05:01.790828 29403 tablet.cc:2369] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:05:01.867682 29263 tablet.cc:2369] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:05:01.925361 29577 raft_consensus.cc:482] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:01.925510 29577 raft_consensus.cc:504] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.925926 29577 leader_election.cc:288] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303), 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303)
I20260927 07:05:01.928972 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.926182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35303 (local address 127.21.2.65:40111)
0927 07:05:01.926322 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.926335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.926403 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.926580 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.926581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.926583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:01.926669 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.926671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.927340 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.927342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.928150 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.928152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.928700 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.928771 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.928775 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.928851 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.928893 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260927 07:05:01.928970 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.926288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:40111 (local address 127.21.2.67:35303)
0927 07:05:01.926334 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.926336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.926400 (+ 64us) server_negotiation.cc:407] Connection header received
0927 07:05:01.926494 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.926495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.926507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.926573 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:01.926759 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.927244 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928251 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.928602 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928698 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.928709 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.928868 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.928907 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:05:01.929170 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.926486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:39415 (local address 127.21.2.66:37303)
0927 07:05:01.926553 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.926555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.926590 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:05:01.926665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.926666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.926684 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.926771 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:01.926924 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.927461 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928408 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.928776 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.928883 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.929059 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.929103 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:05:01.929183 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.926394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.21.2.65:39415)
0927 07:05:01.926542 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:01.926552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:01.926602 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:01.926775 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:01.926776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:01.926778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:01.926842 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.926844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.927562 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.927563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.928297 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:01.928299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.928877 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.928878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:01.928961 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.928967 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:05:01.929053 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 07:05:01.929094 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260927 07:05:01.929741 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfca716b64eb468a9042ee83f384ae2c" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188" is_pre_election: true
I20260927 07:05:01.929760 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfca716b64eb468a9042ee83f384ae2c" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224" is_pre_election: true
I20260927 07:05:01.929903 29496 raft_consensus.cc:2454] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:01.929919 29356 raft_consensus.cc:2454] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:01.930150 29150 leader_election.cc:302] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8602f2ee9a2a44e18f5e1a574b03fe95, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:01.930295 29577 raft_consensus.cc:2781] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:01.930361 29577 raft_consensus.cc:482] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:01.930389 29577 raft_consensus.cc:3037] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:01.931180 29577 raft_consensus.cc:504] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.931339 29577 leader_election.cc:288] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Requested vote from peers ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303), 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303)
I20260927 07:05:01.931583 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfca716b64eb468a9042ee83f384ae2c" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188"
I20260927 07:05:01.931694 29496 raft_consensus.cc:3037] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:01.932389 29496 raft_consensus.cc:2454] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:01.932545 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfca716b64eb468a9042ee83f384ae2c" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224"
I20260927 07:05:01.932611 29150 leader_election.cc:302] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8602f2ee9a2a44e18f5e1a574b03fe95, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:01.932641 29356 raft_consensus.cc:3037] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:01.932722 29577 raft_consensus.cc:2781] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:01.932833 29577 raft_consensus.cc:686] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Becoming Leader. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Running, Role: LEADER
I20260927 07:05:01.933096 29577 consensus_queue.cc:237] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } }
I20260927 07:05:01.933554 29356 raft_consensus.cc:2454] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:01.933907 29058 catalog_manager.cc:5495] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } health_report { overall_health: UNKNOWN } } }
W20260927 07:05:01.940969 29543 tablet.cc:2369] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:05:01.974305 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:01.971911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.21.2.65:34531)
0927 07:05:01.971972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:01.971973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:01.971982 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:01.972025 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:01.972025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:01.972045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:01.972115 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:01.972353 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.972829 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.973579 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:01.973762 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:01.973812 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:01.973951 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:01.974050 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:01.974106 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:01.974224 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 07:05:01.974244 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:05:01.976441 29496 raft_consensus.cc:1261] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:01.976440 29356 raft_consensus.cc:1261] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:01.976848 29581 consensus_queue.cc:1035] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:01.977000 29577 consensus_queue.cc:1035] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:01.979419 29584 mvcc.cc:205] Tried to move back new op lower bound from 7333858107292160000 to 7333858107118727168. Current Snapshot: MvccSnapshot[applied={T|T < 7333858107292160000}]
I20260927 07:05:01.979518 29585 mvcc.cc:205] Tried to move back new op lower bound from 7333858107292160000 to 7333858107118727168. Current Snapshot: MvccSnapshot[applied={T|T < 7333858107292160000}]
I20260927 07:05:01.980029 29586 mvcc.cc:205] Tried to move back new op lower bound from 7333858107292160000 to 7333858107118727168. Current Snapshot: MvccSnapshot[applied={T|T < 7333858107292160000}]
I20260927 07:05:01.982086 29058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:01.982187 29058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:01.982950 29058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Sending DeleteTablet for 3 replicas of tablet bfca716b64eb468a9042ee83f384ae2c
I20260927 07:05:01.983278 29196 tablet_service.cc:1514] Processing DeleteTablet for tablet bfca716b64eb468a9042ee83f384ae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:01 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 07:05:01.983433 29476 tablet_service.cc:1514] Processing DeleteTablet for tablet bfca716b64eb468a9042ee83f384ae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:01 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50074
I20260927 07:05:01.983584 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet bfca716b64eb468a9042ee83f384ae2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:01 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50798
I20260927 07:05:01.983628 29593 tablet_replica.cc:330] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: stopping tablet replica
I20260927 07:05:01.983754 29593 raft_consensus.cc:2229] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:05:01.983850 29593 raft_consensus.cc:2258] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:01.983935 29595 tablet_replica.cc:330] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: stopping tablet replica
I20260927 07:05:01.984022 29058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 07:05:01.984098 29595 raft_consensus.cc:2229] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 07:05:01.984174 29058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:01.984207 29595 raft_consensus.cc:2258] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:01.984359 29593 ts_tablet_manager.cc:1905] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:01.984634 29595 ts_tablet_manager.cc:1905] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:01.986075 29319 tablet_service.cc:1467] Processing CreateTablet for tablet df6528257ff84a45be3a228d05aacb3a (DEFAULT_TABLE table=test-table [id=a43ccd980e4a4681bbefe10862c38189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.986142 29594 tablet_replica.cc:330] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: stopping tablet replica
I20260927 07:05:01.986204 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6528257ff84a45be3a228d05aacb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.986243 29594 raft_consensus.cc:2229] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:01.986303 29476 tablet_service.cc:1467] Processing CreateTablet for tablet df6528257ff84a45be3a228d05aacb3a (DEFAULT_TABLE table=test-table [id=a43ccd980e4a4681bbefe10862c38189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.986351 29594 raft_consensus.cc:2258] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:01.986400 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6528257ff84a45be3a228d05aacb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.986845 29594 ts_tablet_manager.cc:1905] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:01.987406 29595 ts_tablet_manager.cc:1918] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:01.987469 29572 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Bootstrap starting.
I20260927 07:05:01.987478 29595 log.cc:1196] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/bfca716b64eb468a9042ee83f384ae2c
I20260927 07:05:01.987774 29196 tablet_service.cc:1467] Processing CreateTablet for tablet df6528257ff84a45be3a228d05aacb3a (DEFAULT_TABLE table=test-table [id=a43ccd980e4a4681bbefe10862c38189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:01.987872 29572 tablet_bootstrap.cc:654] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.987895 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6528257ff84a45be3a228d05aacb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:01.988016 29595 ts_tablet_manager.cc:1939] T bfca716b64eb468a9042ee83f384ae2c P 71ef83a2beb84430bb771cc9fbb22224: Deleting consensus metadata
I20260927 07:05:01.988147 29593 ts_tablet_manager.cc:1918] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:01.988212 29593 log.cc:1196] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/bfca716b64eb468a9042ee83f384ae2c
I20260927 07:05:01.988400 29046 catalog_manager.cc:4841] TS 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303): tablet bfca716b64eb468a9042ee83f384ae2c (table test-table [id=ff931ea0c038457eb7c8ab5ff691497e]) successfully deleted
I20260927 07:05:01.988456 29593 ts_tablet_manager.cc:1939] T bfca716b64eb468a9042ee83f384ae2c P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting consensus metadata
I20260927 07:05:01.988893 29044 catalog_manager.cc:4841] TS 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531): tablet bfca716b64eb468a9042ee83f384ae2c (table test-table [id=ff931ea0c038457eb7c8ab5ff691497e]) successfully deleted
I20260927 07:05:01.989230 29571 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Bootstrap starting.
I20260927 07:05:01.989259 29570 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Bootstrap starting.
I20260927 07:05:01.989651 29571 tablet_bootstrap.cc:654] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.989651 29570 tablet_bootstrap.cc:654] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:01.989675 29594 ts_tablet_manager.cc:1918] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:01.989779 29594 log.cc:1196] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/bfca716b64eb468a9042ee83f384ae2c
I20260927 07:05:01.990027 29594 ts_tablet_manager.cc:1939] T bfca716b64eb468a9042ee83f384ae2c P ce7eaf9ca10748cf9c53f686700c5188: Deleting consensus metadata
I20260927 07:05:01.990379 29046 catalog_manager.cc:4841] TS ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303): tablet bfca716b64eb468a9042ee83f384ae2c (table test-table [id=ff931ea0c038457eb7c8ab5ff691497e]) successfully deleted
I20260927 07:05:01.990533 29570 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: No bootstrap required, opened a new log
I20260927 07:05:01.990588 29570 ts_tablet_manager.cc:1397] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:01.990772 29570 raft_consensus.cc:348] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.990844 29570 raft_consensus.cc:374] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.990859 29570 raft_consensus.cc:729] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.990923 29570 consensus_queue.cc:260] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.991012 29570 ts_tablet_manager.cc:1428] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:01.991461 29571 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: No bootstrap required, opened a new log
I20260927 07:05:01.991511 29571 ts_tablet_manager.cc:1397] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:05:01.991653 29571 raft_consensus.cc:348] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.991771 29571 raft_consensus.cc:374] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.991794 29571 raft_consensus.cc:729] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.991835 29572 tablet_bootstrap.cc:492] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: No bootstrap required, opened a new log
I20260927 07:05:01.991837 29571 consensus_queue.cc:260] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.991876 29572 ts_tablet_manager.cc:1397] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 07:05:01.991915 29571 ts_tablet_manager.cc:1428] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:01.992025 29572 raft_consensus.cc:348] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.992095 29572 raft_consensus.cc:374] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:01.992120 29572 raft_consensus.cc:729] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Initialized, Role: FOLLOWER
I20260927 07:05:01.992184 29572 consensus_queue.cc:260] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:01.992261 29572 ts_tablet_manager.cc:1428] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:02.199523 29576 raft_consensus.cc:482] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:02.199648 29576 raft_consensus.cc:504] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.200030 29576 leader_election.cc:288] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531), 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303)
I20260927 07:05:02.203130 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.200228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34531 (local address 127.21.2.67:35393)
0927 07:05:02.200395 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.200406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.200496 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.200699 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.200700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.200702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:02.200756 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.200758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.201412 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.201413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.202265 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.202268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.202840 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.202930 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.202936 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.202992 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.203033 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":57,"threads_started":1}
I20260927 07:05:02.203153 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.200355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35393 (local address 127.21.2.65:34531)
0927 07:05:02.200403 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.200405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.200464 (+ 59us) server_negotiation.cc:407] Connection header received
0927 07:05:02.200595 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.200596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.200606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.200652 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:02.200834 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.201328 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202356 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.202758 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202812 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.202821 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.203008 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.203048 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:05:02.203279 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.200357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:37391 (local address 127.21.2.66:37303)
0927 07:05:02.200444 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.200446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.200482 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:05:02.200569 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.200570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.200580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.200669 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:02.200903 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.201460 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202478 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.202871 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202941 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.202951 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.203148 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.203194 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:05:02.203249 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.200268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.21.2.67:37391)
0927 07:05:02.200412 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.200423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.200495 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.200675 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.200676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.200678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:02.200762 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.200763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.201548 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.201549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.202392 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.202394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.202962 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.202962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.203057 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.203061 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.203120 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.203158 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260927 07:05:02.203552 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6528257ff84a45be3a228d05aacb3a" candidate_uuid: "ce7eaf9ca10748cf9c53f686700c5188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224" is_pre_election: true
I20260927 07:05:02.203653 29356 raft_consensus.cc:2454] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7eaf9ca10748cf9c53f686700c5188 in term 0.
I20260927 07:05:02.203572 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6528257ff84a45be3a228d05aacb3a" candidate_uuid: "ce7eaf9ca10748cf9c53f686700c5188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" is_pre_election: true
I20260927 07:05:02.203716 29216 raft_consensus.cc:2454] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7eaf9ca10748cf9c53f686700c5188 in term 0.
I20260927 07:05:02.203842 29430 leader_election.cc:302] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ef83a2beb84430bb771cc9fbb22224, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.204005 29576 raft_consensus.cc:2781] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:02.204051 29576 raft_consensus.cc:482] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:02.204066 29576 raft_consensus.cc:3037] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.204756 29576 raft_consensus.cc:504] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.204912 29576 leader_election.cc:288] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [CANDIDATE]: Term 1 election: Requested vote from peers 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531), 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303)
I20260927 07:05:02.205071 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6528257ff84a45be3a228d05aacb3a" candidate_uuid: "ce7eaf9ca10748cf9c53f686700c5188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224"
I20260927 07:05:02.205099 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6528257ff84a45be3a228d05aacb3a" candidate_uuid: "ce7eaf9ca10748cf9c53f686700c5188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95"
I20260927 07:05:02.205192 29356 raft_consensus.cc:3037] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.205186 29216 raft_consensus.cc:3037] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.205797 29356 raft_consensus.cc:2454] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7eaf9ca10748cf9c53f686700c5188 in term 1.
I20260927 07:05:02.205976 29216 raft_consensus.cc:2454] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7eaf9ca10748cf9c53f686700c5188 in term 1.
I20260927 07:05:02.205996 29430 leader_election.cc:302] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ef83a2beb84430bb771cc9fbb22224, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.206094 29576 raft_consensus.cc:2781] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:02.206219 29576 raft_consensus.cc:686] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 1 LEADER]: Becoming Leader. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Running, Role: LEADER
I20260927 07:05:02.206321 29576 consensus_queue.cc:237] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.207015 29058 catalog_manager.cc:5495] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "ce7eaf9ca10748cf9c53f686700c5188" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } health_report { overall_health: HEALTHY } } }
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492701, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.126@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:02.247270 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.240161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.21.2.126:46793)
0927 07:05:02.240324 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.240326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.240336 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:05:02.240358 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.240359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.240368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.240432 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:02.240862 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.241377 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.242139 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.242346 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.242402 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.244783 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:02.244784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:02.244784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:02.244791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:02.246281 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:02.246611 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:02.246612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:02.246612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:02.246673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:02.246841 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:02.246841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:02.246842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:02.246942 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:02.246997 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.247122 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.247197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260927 07:05:02.247411 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.240063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:46793 (local address 127.0.0.1:46996)
0927 07:05:02.240221 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.240233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.240313 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.240456 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.240456 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.240745 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:02.240790 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.240792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.241453 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.241454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.242000 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.242002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.242415 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.242416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.242460 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.243067 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:02.243071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:02.244663 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:02.246351 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:02.246357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:02.246359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:02.246544 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:02.246732 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:02.246732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:02.246733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:02.246786 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:02.247002 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:02.247008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:02.247050 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.247104 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.247246 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":53,"thread_start_us":81,"threads_started":1}
I20260927 07:05:02.249322 29058 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:46996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:05:02.249522 29058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:02.250988 29196 tablet_service.cc:1467] Processing CreateTablet for tablet ce6888d146e74aa0bc80326034dc8468 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=794c23908e5548f0b1439fd8a41231ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:05:02.250984 29319 tablet_service.cc:1467] Processing CreateTablet for tablet ce6888d146e74aa0bc80326034dc8468 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=794c23908e5548f0b1439fd8a41231ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:05:02.251000 29476 tablet_service.cc:1467] Processing CreateTablet for tablet ce6888d146e74aa0bc80326034dc8468 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=794c23908e5548f0b1439fd8a41231ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:05:02.251139 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6888d146e74aa0bc80326034dc8468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.251148 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6888d146e74aa0bc80326034dc8468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.251160 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6888d146e74aa0bc80326034dc8468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.252526 29571 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: Bootstrap starting.
I20260927 07:05:02.252688 29570 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: Bootstrap starting.
I20260927 07:05:02.252686 29572 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: Bootstrap starting.
I20260927 07:05:02.252862 29571 tablet_bootstrap.cc:654] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.253105 29570 tablet_bootstrap.cc:654] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.253073 29572 tablet_bootstrap.cc:654] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.253482 29571 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: No bootstrap required, opened a new log
I20260927 07:05:02.253535 29571 ts_tablet_manager.cc:1397] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:05:02.253681 29571 raft_consensus.cc:348] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.253747 29570 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: No bootstrap required, opened a new log
I20260927 07:05:02.253764 29571 raft_consensus.cc:374] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.253787 29571 raft_consensus.cc:729] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.253790 29570 ts_tablet_manager.cc:1397] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:02.253835 29571 consensus_queue.cc:260] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.253943 29571 ts_tablet_manager.cc:1428] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 07:05:02.253947 29572 tablet_bootstrap.cc:492] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: No bootstrap required, opened a new log
I20260927 07:05:02.253933 29570 raft_consensus.cc:348] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.253990 29572 ts_tablet_manager.cc:1397] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:02.254011 29570 raft_consensus.cc:374] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.254046 29570 raft_consensus.cc:729] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.254098 29570 consensus_queue.cc:260] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.254154 29572 raft_consensus.cc:348] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.254194 29570 ts_tablet_manager.cc:1428] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:02.254171 29581 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.254222 29572 raft_consensus.cc:374] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.254243 29572 raft_consensus.cc:729] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.254246 29581 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.254297 29572 consensus_queue.cc:260] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.254318 29576 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.254386 29576 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.254405 29572 ts_tablet_manager.cc:1428] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:02.254521 29578 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.254596 29578 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.427977 29578 raft_consensus.cc:482] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:02.428102 29578 raft_consensus.cc:504] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.428508 29578 leader_election.cc:288] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.431761 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.428802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:33481 (local address 127.21.2.65:34531)
0927 07:05:02.428857 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.428861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.428964 (+ 103us) server_negotiation.cc:407] Connection header received
0927 07:05:02.429051 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.429052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.429064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.429138 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:02.429322 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.429900 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.430963 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.431349 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.431470 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.431482 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.431632 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.431668 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:05:02.431761 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.428858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:48965 (local address 127.21.2.67:35303)
0927 07:05:02.428905 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.428908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.429007 (+ 99us) server_negotiation.cc:407] Connection header received
0927 07:05:02.429051 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.429052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.429061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.429146 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:02.429335 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.429910 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.430960 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.431331 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.431458 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.431470 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.431629 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.431669 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:05:02.431784 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.428778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35303 (local address 127.21.2.66:48965)
0927 07:05:02.428901 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.428913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.428985 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.429153 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.429154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.429156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:02.429253 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.429254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.430017 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.430018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.430857 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.430859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.431461 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.431462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.431546 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.431549 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.431622 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.431669 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":39,"thread_start_us":61,"threads_started":1}
I20260927 07:05:02.431790 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.428710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34531 (local address 127.21.2.66:33481)
0927 07:05:02.428882 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.428893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.428981 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.429156 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.429157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.429159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:02.429239 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.429241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.429985 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.429986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.430864 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.430866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.431470 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.431471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.431541 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.431547 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.431625 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.431668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":40,"thread_start_us":87,"threads_started":1}
I20260927 07:05:02.432013 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6888d146e74aa0bc80326034dc8468" candidate_uuid: "71ef83a2beb84430bb771cc9fbb22224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188" is_pre_election: true
I20260927 07:05:02.432117 29496 raft_consensus.cc:2454] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ef83a2beb84430bb771cc9fbb22224 in term 0.
I20260927 07:05:02.432101 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6888d146e74aa0bc80326034dc8468" candidate_uuid: "71ef83a2beb84430bb771cc9fbb22224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" is_pre_election: true
I20260927 07:05:02.432193 29216 raft_consensus.cc:2454] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71ef83a2beb84430bb771cc9fbb22224 in term 0.
I20260927 07:05:02.432348 29290 leader_election.cc:302] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ef83a2beb84430bb771cc9fbb22224, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.432518 29578 raft_consensus.cc:2781] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:02.432566 29578 raft_consensus.cc:482] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:02.432588 29578 raft_consensus.cc:3037] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.433303 29578 raft_consensus.cc:504] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.433470 29578 leader_election.cc:288] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [CANDIDATE]: Term 1 election: Requested vote from peers 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.433665 29216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6888d146e74aa0bc80326034dc8468" candidate_uuid: "71ef83a2beb84430bb771cc9fbb22224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95"
I20260927 07:05:02.433745 29216 raft_consensus.cc:3037] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.433727 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6888d146e74aa0bc80326034dc8468" candidate_uuid: "71ef83a2beb84430bb771cc9fbb22224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188"
I20260927 07:05:02.433818 29496 raft_consensus.cc:3037] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.434428 29216 raft_consensus.cc:2454] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ef83a2beb84430bb771cc9fbb22224 in term 1.
I20260927 07:05:02.434548 29496 raft_consensus.cc:2454] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71ef83a2beb84430bb771cc9fbb22224 in term 1.
I20260927 07:05:02.434618 29288 leader_election.cc:302] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ef83a2beb84430bb771cc9fbb22224, 8602f2ee9a2a44e18f5e1a574b03fe95; no voters:
I20260927 07:05:02.434710 29578 raft_consensus.cc:2781] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:02.434841 29578 raft_consensus.cc:686] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 LEADER]: Becoming Leader. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Running, Role: LEADER
I20260927 07:05:02.434935 29578 consensus_queue.cc:237] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.435299 29615 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: TxnStatusTablet state changed. Reason: New leader 71ef83a2beb84430bb771cc9fbb22224. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.435431 29615 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:05:02.435626 29617 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:05:02.435720 29617 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:05:02.435680 29058 catalog_manager.cc:5495] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } health_report { overall_health: UNKNOWN } } }
I20260927 07:05:02.484223 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.481266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.0.0.1:50806)
0927 07:05:02.481347 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.481361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.481415 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.481613 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.481614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.481615 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:05:02.481673 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.481676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.482617 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.482618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.483210 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.483212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.483707 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.483708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.483758 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.483774 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.484063 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.484070 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.484105 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.484159 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:05:02.484213 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.481370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.21.2.66:37303)
0927 07:05:02.481442 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.481444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.481455 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:02.481504 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.481505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.481528 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.481592 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.481756 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.482518 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.483304 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.483627 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.483699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.483871 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.483996 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.484070 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.484109 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.484129 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:05:02.485424 29496 raft_consensus.cc:1261] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Refusing update from remote peer 71ef83a2beb84430bb771cc9fbb22224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.485531 29216 raft_consensus.cc:1261] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Refusing update from remote peer 71ef83a2beb84430bb771cc9fbb22224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.485682 29615 consensus_queue.cc:1035] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.485790 29578 consensus_queue.cc:1035] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.486330 29576 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: TxnStatusTablet state changed. Reason: New leader 71ef83a2beb84430bb771cc9fbb22224. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.486433 29576 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.486475 29581 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: TxnStatusTablet state changed. Reason: New leader 71ef83a2beb84430bb771cc9fbb22224. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.486572 29581 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.487865 29576 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.487950 29576 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P ce7eaf9ca10748cf9c53f686700c5188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.488198 29581 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.488289 29581 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 8602f2ee9a2a44e18f5e1a574b03fe95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:05:02.488539 29615 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.488637 29615 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:05:02.488842 29627 tablet_replica.cc:439] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71ef83a2beb84430bb771cc9fbb22224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } } }
I20260927 07:05:02.488948 29627 tablet_replica.cc:442] T ce6888d146e74aa0bc80326034dc8468 P 71ef83a2beb84430bb771cc9fbb22224: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:05:02.492544 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.489910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.21.2.67:35303)
0927 07:05:02.489952 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.489954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.489962 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:02.490019 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.490020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.490038 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.490113 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.490245 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.490798 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.491767 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.491972 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.492059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.492209 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.492313 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.492394 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.492448 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.492467 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:05:02.497725 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.494849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.21.2.67:41629)
0927 07:05:02.494981 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.494992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.495053 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.495174 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.495175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.495176 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:05:02.495224 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.495226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.496164 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.496164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.496793 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.496796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.497292 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.497294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.497365 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.497379 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.497588 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.497590 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.497622 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.497657 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260927 07:05:02.497730 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.494920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:41629 (local address 127.21.2.66:37303)
0927 07:05:02.494968 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.494969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.495065 (+ 96us) server_negotiation.cc:407] Connection header received
0927 07:05:02.495092 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.495092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.495103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.495174 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.495266 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.496068 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.496877 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.497166 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.497264 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.497481 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.497515 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.497576 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.497638 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.497658 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:05:02.500079 29356 raft_consensus.cc:1261] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Refusing update from remote peer ce7eaf9ca10748cf9c53f686700c5188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.500123 29216 raft_consensus.cc:1261] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Refusing update from remote peer ce7eaf9ca10748cf9c53f686700c5188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.500346 29601 consensus_queue.cc:1035] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.500545 29601 consensus_queue.cc:1035] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.509150 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.506358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35303 (local address 127.21.2.66:33169)
0927 07:05:02.506508 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.506519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.506603 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.506759 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.506760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.506762 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:05:02.506830 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.506832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.507623 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.507624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.508223 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.508225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.508707 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.508708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.508785 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.508809 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.509005 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.509006 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.509037 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.509078 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260927 07:05:02.509140 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.506396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:33169 (local address 127.21.2.67:35303)
0927 07:05:02.506435 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.506437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.506575 (+ 138us) server_negotiation.cc:407] Connection header received
0927 07:05:02.506662 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.506663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.506672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.506715 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.506913 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.507454 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.508332 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.508610 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.508671 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.508898 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.508937 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.509006 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.509042 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.509066 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:05:02.516533 29059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.516665 29059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.517278 29059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Sending DeleteTablet for 3 replicas of tablet df6528257ff84a45be3a228d05aacb3a
I20260927 07:05:02.517613 29196 tablet_service.cc:1514] Processing DeleteTablet for tablet df6528257ff84a45be3a228d05aacb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 07:05:02.517655 29476 tablet_service.cc:1514] Processing DeleteTablet for tablet df6528257ff84a45be3a228d05aacb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50074
I20260927 07:05:02.517834 29642 tablet_replica.cc:330] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: stopping tablet replica
I20260927 07:05:02.517887 29642 raft_consensus.cc:2229] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:05:02.517968 29642 raft_consensus.cc:2258] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.518158 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet df6528257ff84a45be3a228d05aacb3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50798
I20260927 07:05:02.518409 29641 tablet_replica.cc:330] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: stopping tablet replica
I20260927 07:05:02.518500 29641 raft_consensus.cc:2229] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.518587 29641 raft_consensus.cc:2258] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.518900 29641 ts_tablet_manager.cc:1905] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.521013 29641 ts_tablet_manager.cc:1918] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:05:02.521200 29643 tablet_replica.cc:330] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: stopping tablet replica
I20260927 07:05:02.521243 29642 ts_tablet_manager.cc:1905] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.521344 29643 raft_consensus.cc:2229] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.521441 29643 raft_consensus.cc:2258] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.521571 29641 log.cc:1196] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/df6528257ff84a45be3a228d05aacb3a
I20260927 07:05:02.521775 29643 ts_tablet_manager.cc:1905] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.523223 29642 ts_tablet_manager.cc:1918] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:05:02.523315 29642 log.cc:1196] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/df6528257ff84a45be3a228d05aacb3a
I20260927 07:05:02.523698 29642 ts_tablet_manager.cc:1939] T df6528257ff84a45be3a228d05aacb3a P ce7eaf9ca10748cf9c53f686700c5188: Deleting consensus metadata
I20260927 07:05:02.524142 29046 catalog_manager.cc:4841] TS ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303): tablet df6528257ff84a45be3a228d05aacb3a (table test-table [id=a43ccd980e4a4681bbefe10862c38189]) successfully deleted
I20260927 07:05:02.524907 29643 ts_tablet_manager.cc:1918] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:05:02.525286 29641 ts_tablet_manager.cc:1939] T df6528257ff84a45be3a228d05aacb3a P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting consensus metadata
I20260927 07:05:02.525619 29643 log.cc:1196] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/df6528257ff84a45be3a228d05aacb3a
I20260927 07:05:02.525691 29044 catalog_manager.cc:4841] TS 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531): tablet df6528257ff84a45be3a228d05aacb3a (table test-table [id=a43ccd980e4a4681bbefe10862c38189]) successfully deleted
I20260927 07:05:02.525949 29643 ts_tablet_manager.cc:1939] T df6528257ff84a45be3a228d05aacb3a P 71ef83a2beb84430bb771cc9fbb22224: Deleting consensus metadata
I20260927 07:05:02.526299 29046 catalog_manager.cc:4841] TS 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303): tablet df6528257ff84a45be3a228d05aacb3a (table test-table [id=a43ccd980e4a4681bbefe10862c38189]) successfully deleted
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:05:02.551925 29609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:02.546327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.21.2.126:46793)
0927 07:05:02.546480 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.546482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.546528 (+ 46us) server_negotiation.cc:407] Connection header received
0927 07:05:02.546653 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.546654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.546669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.546717 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:02.547360 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.547901 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.548667 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.548881 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.548962 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.551873 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47002: BlockingRecv error: recv got EOF from 127.0.0.1:47002 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM
I20260927 07:05:02.560277 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.553134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.21.2.126:46793)
0927 07:05:02.553172 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.553174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.553231 (+ 57us) server_negotiation.cc:407] Connection header received
0927 07:05:02.553296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.553297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.553306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.553356 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:02.553821 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.554286 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.554969 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.555172 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.555254 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.557619 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:02.557621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:02.557621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:02.557629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:02.559094 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:02.559487 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:02.559489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:02.559489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:02.559548 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:02.559751 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:02.559752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:02.559752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:02.559871 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:02.559989 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.560080 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.560162 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:05:02.562135 29059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:05:02.562330 29059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:02.563755 29196 tablet_service.cc:1467] Processing CreateTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 (DEFAULT_TABLE table=test-table [id=31fa462ea9864fa29c2456247f9782f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.563822 29319 tablet_service.cc:1467] Processing CreateTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 (DEFAULT_TABLE table=test-table [id=31fa462ea9864fa29c2456247f9782f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.563894 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f411cc0fca041df9a2652fe0feb7085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.563939 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f411cc0fca041df9a2652fe0feb7085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.565228 29571 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Bootstrap starting.
I20260927 07:05:02.565270 29476 tablet_service.cc:1467] Processing CreateTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 (DEFAULT_TABLE table=test-table [id=31fa462ea9864fa29c2456247f9782f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.565382 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f411cc0fca041df9a2652fe0feb7085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.565560 29571 tablet_bootstrap.cc:654] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.566423 29571 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: No bootstrap required, opened a new log
I20260927 07:05:02.566471 29571 ts_tablet_manager.cc:1397] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:05:02.566576 29571 raft_consensus.cc:348] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.566650 29571 raft_consensus.cc:374] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.566673 29571 raft_consensus.cc:729] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.566718 29571 consensus_queue.cc:260] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.566893 29572 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Bootstrap starting.
I20260927 07:05:02.567098 29571 ts_tablet_manager.cc:1428] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:05:02.567294 29572 tablet_bootstrap.cc:654] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.567692 29570 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Bootstrap starting.
I20260927 07:05:02.568293 29570 tablet_bootstrap.cc:654] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.569060 29572 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: No bootstrap required, opened a new log
I20260927 07:05:02.569144 29572 ts_tablet_manager.cc:1397] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:05:02.569301 29572 raft_consensus.cc:348] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.569393 29572 raft_consensus.cc:374] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.569422 29572 raft_consensus.cc:729] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.569463 29572 consensus_queue.cc:260] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.569556 29572 ts_tablet_manager.cc:1428] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:02.570042 29570 tablet_bootstrap.cc:492] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: No bootstrap required, opened a new log
I20260927 07:05:02.570096 29570 ts_tablet_manager.cc:1397] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:05:02.570242 29570 raft_consensus.cc:348] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.570336 29570 raft_consensus.cc:374] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.570363 29570 raft_consensus.cc:729] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.570403 29570 consensus_queue.cc:260] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.570519 29570 ts_tablet_manager.cc:1428] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:02.572903 29581 raft_consensus.cc:482] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:02.572973 29581 raft_consensus.cc:504] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.573100 29581 leader_election.cc:288] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.573284 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f411cc0fca041df9a2652fe0feb7085" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188" is_pre_election: true
I20260927 07:05:02.573354 29496 raft_consensus.cc:2454] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:02.573475 29150 leader_election.cc:302] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8602f2ee9a2a44e18f5e1a574b03fe95, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.573603 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f411cc0fca041df9a2652fe0feb7085" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224" is_pre_election: true
I20260927 07:05:02.573693 29356 raft_consensus.cc:2454] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:02.573868 29581 raft_consensus.cc:2781] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:02.573908 29581 raft_consensus.cc:482] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:02.573936 29581 raft_consensus.cc:3037] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.574551 29581 raft_consensus.cc:504] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.574692 29581 leader_election.cc:288] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Requested vote from peers 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.574889 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f411cc0fca041df9a2652fe0feb7085" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224"
I20260927 07:05:02.574885 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f411cc0fca041df9a2652fe0feb7085" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188"
I20260927 07:05:02.574949 29356 raft_consensus.cc:3037] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.574949 29496 raft_consensus.cc:3037] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.575698 29496 raft_consensus.cc:2454] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:02.575768 29356 raft_consensus.cc:2454] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:02.575911 29150 leader_election.cc:302] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8602f2ee9a2a44e18f5e1a574b03fe95, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.576064 29581 raft_consensus.cc:2781] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:02.576195 29581 raft_consensus.cc:686] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Becoming Leader. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Running, Role: LEADER
I20260927 07:05:02.576252 29581 consensus_queue.cc:237] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.576856 29059 catalog_manager.cc:5495] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } health_report { overall_health: UNKNOWN } } }
I20260927 07:05:02.582862 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.580414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.21.2.65:34531)
0927 07:05:02.580454 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.580456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.580485 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:05:02.580553 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.580553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.580564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.580632 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.580745 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.581300 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.582143 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.582355 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.582433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.582589 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.582624 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.582695 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.582754 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.582777 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:05:02.583621 29356 raft_consensus.cc:1261] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.583933 29496 raft_consensus.cc:1261] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.583912 29659 consensus_queue.cc:1035] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.584828 29659 consensus_queue.cc:1035] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.586409 29059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.586514 29059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.587231 29059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Sending DeleteTablet for 3 replicas of tablet 2f411cc0fca041df9a2652fe0feb7085
I20260927 07:05:02.587491 29196 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 07:05:02.587575 29641 tablet_replica.cc:330] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: stopping tablet replica
I20260927 07:05:02.587623 29641 raft_consensus.cc:2229] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:05:02.587679 29641 raft_consensus.cc:2258] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.587940 29476 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50074
I20260927 07:05:02.588024 29642 tablet_replica.cc:330] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: stopping tablet replica
I20260927 07:05:02.588076 29642 raft_consensus.cc:2229] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.588119 29642 raft_consensus.cc:2258] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.588558 29641 ts_tablet_manager.cc:1905] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.588557 29642 ts_tablet_manager.cc:1905] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.590170 29642 ts_tablet_manager.cc:1918] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:02.590202 29641 ts_tablet_manager.cc:1918] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:02.590229 29642 log.cc:1196] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/2f411cc0fca041df9a2652fe0feb7085
I20260927 07:05:02.590261 29641 log.cc:1196] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/2f411cc0fca041df9a2652fe0feb7085
I20260927 07:05:02.590189 29058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:05:02.590340 29058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:02.590448 29642 ts_tablet_manager.cc:1939] T 2f411cc0fca041df9a2652fe0feb7085 P ce7eaf9ca10748cf9c53f686700c5188: Deleting consensus metadata
I20260927 07:05:02.590502 29641 ts_tablet_manager.cc:1939] T 2f411cc0fca041df9a2652fe0feb7085 P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting consensus metadata
I20260927 07:05:02.590971 29044 catalog_manager.cc:4841] TS 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531): tablet 2f411cc0fca041df9a2652fe0feb7085 (table test-table [id=31fa462ea9864fa29c2456247f9782f9]) successfully deleted
I20260927 07:05:02.591066 29046 catalog_manager.cc:4841] TS ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303): tablet 2f411cc0fca041df9a2652fe0feb7085 (table test-table [id=31fa462ea9864fa29c2456247f9782f9]) successfully deleted
I20260927 07:05:02.591197 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f411cc0fca041df9a2652fe0feb7085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50798
I20260927 07:05:02.591415 29643 tablet_replica.cc:330] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: stopping tablet replica
I20260927 07:05:02.591480 29643 raft_consensus.cc:2229] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.591523 29643 pending_rounds.cc:62] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Trying to abort 2 pending ops.
I20260927 07:05:02.591552 29643 pending_rounds.cc:68] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333858109780578304 op_type: WRITE_OP write_request { tablet_id: "2f411cc0fca041df9a2652fe0feb7085" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\312\202\343\325\006\"8\n\ntest-admin\022*\n 31fa462ea9864fa29c2456247f9782f9\020\001\030\001 \001(\001" signature: "<redacted>""u\001\035\347A\311\035\373<\311\rY\357\230\316\\\307\003\003\250y\262\265\317 2\022\340\316\322\376\250\007#J\270\326\222D\202\202\210\211\305rlqf\206\303\310\035\226\347\0230\274\341\224=9O\223t" signing_key_seq_num: 0 } } request_id { client_id: "a646040f15754366a5bf9dea1e8e7ece" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 07:05:02.591795 29643 pending_rounds.cc:68] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333858109752659968 op_type: NO_OP noop_request { }
I20260927 07:05:02.591846 29643 raft_consensus.cc:2866] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 07:05:02.591883 29643 raft_consensus.cc:2258] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.592172 29643 ts_tablet_manager.cc:1905] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.592718 29196 tablet_service.cc:1467] Processing CreateTablet for tablet e0479ade29bc4525891d693d20da6f6d (DEFAULT_TABLE table=test-table [id=ce2c9123eb094543b5363209f88ea890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.592818 29196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0479ade29bc4525891d693d20da6f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.593690 29643 ts_tablet_manager.cc:1918] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:02.593737 29643 log.cc:1196] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/2f411cc0fca041df9a2652fe0feb7085
I20260927 07:05:02.594198 29643 ts_tablet_manager.cc:1939] T 2f411cc0fca041df9a2652fe0feb7085 P 71ef83a2beb84430bb771cc9fbb22224: Deleting consensus metadata
I20260927 07:05:02.594261 29476 tablet_service.cc:1467] Processing CreateTablet for tablet e0479ade29bc4525891d693d20da6f6d (DEFAULT_TABLE table=test-table [id=ce2c9123eb094543b5363209f88ea890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.594355 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0479ade29bc4525891d693d20da6f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.595304 29570 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Bootstrap starting.
I20260927 07:05:02.595503 29319 tablet_service.cc:1467] Processing CreateTablet for tablet e0479ade29bc4525891d693d20da6f6d (DEFAULT_TABLE table=test-table [id=ce2c9123eb094543b5363209f88ea890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:02.595584 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0479ade29bc4525891d693d20da6f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:02.595976 29570 tablet_bootstrap.cc:654] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.596850 29572 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Bootstrap starting.
I20260927 07:05:02.596956 29046 catalog_manager.cc:4841] TS 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303): tablet 2f411cc0fca041df9a2652fe0feb7085 (table test-table [id=31fa462ea9864fa29c2456247f9782f9]) successfully deleted
I20260927 07:05:02.597155 29572 tablet_bootstrap.cc:654] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.597621 29571 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: Bootstrap starting.
I20260927 07:05:02.597950 29571 tablet_bootstrap.cc:654] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:02.598512 29571 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: No bootstrap required, opened a new log
I20260927 07:05:02.598564 29571 ts_tablet_manager.cc:1397] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:05:02.598687 29571 raft_consensus.cc:348] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.598764 29571 raft_consensus.cc:374] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.598786 29571 raft_consensus.cc:729] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.598801 29572 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: No bootstrap required, opened a new log
I20260927 07:05:02.598841 29572 ts_tablet_manager.cc:1397] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:05:02.598827 29571 consensus_queue.cc:260] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.598918 29571 ts_tablet_manager.cc:1428] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:02.598943 29572 raft_consensus.cc:348] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.599009 29572 raft_consensus.cc:374] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.599037 29572 raft_consensus.cc:729] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ef83a2beb84430bb771cc9fbb22224, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.599079 29572 consensus_queue.cc:260] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.599150 29572 ts_tablet_manager.cc:1428] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:02.599977 29570 tablet_bootstrap.cc:492] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: No bootstrap required, opened a new log
I20260927 07:05:02.600051 29570 ts_tablet_manager.cc:1397] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 07:05:02.600190 29570 raft_consensus.cc:348] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.600274 29570 raft_consensus.cc:374] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:02.600303 29570 raft_consensus.cc:729] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7eaf9ca10748cf9c53f686700c5188, State: Initialized, Role: FOLLOWER
I20260927 07:05:02.600347 29570 consensus_queue.cc:260] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.600435 29570 ts_tablet_manager.cc:1428] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:02.654475 29581 raft_consensus.cc:482] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:02.654588 29581 raft_consensus.cc:504] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.654796 29581 leader_election.cc:288] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.655012 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0479ade29bc4525891d693d20da6f6d" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188" is_pre_election: true
I20260927 07:05:02.655010 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0479ade29bc4525891d693d20da6f6d" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224" is_pre_election: true
I20260927 07:05:02.655114 29356 raft_consensus.cc:2454] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:02.655129 29496 raft_consensus.cc:2454] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 0.
I20260927 07:05:02.655328 29150 leader_election.cc:302] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8602f2ee9a2a44e18f5e1a574b03fe95, ce7eaf9ca10748cf9c53f686700c5188; no voters:
I20260927 07:05:02.655452 29581 raft_consensus.cc:2781] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:02.655495 29581 raft_consensus.cc:482] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:02.655520 29581 raft_consensus.cc:3037] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.656237 29581 raft_consensus.cc:504] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.656365 29581 leader_election.cc:288] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Requested vote from peers 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303), ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303)
I20260927 07:05:02.656517 29356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0479ade29bc4525891d693d20da6f6d" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ef83a2beb84430bb771cc9fbb22224"
I20260927 07:05:02.656601 29356 raft_consensus.cc:3037] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.656574 29496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0479ade29bc4525891d693d20da6f6d" candidate_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7eaf9ca10748cf9c53f686700c5188"
I20260927 07:05:02.656667 29496 raft_consensus.cc:3037] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:02.657263 29356 raft_consensus.cc:2454] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:02.657425 29496 raft_consensus.cc:2454] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8602f2ee9a2a44e18f5e1a574b03fe95 in term 1.
I20260927 07:05:02.657444 29150 leader_election.cc:302] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71ef83a2beb84430bb771cc9fbb22224, 8602f2ee9a2a44e18f5e1a574b03fe95; no voters:
I20260927 07:05:02.657574 29581 raft_consensus.cc:2781] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:02.657629 29581 raft_consensus.cc:686] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Becoming Leader. State: Replica: 8602f2ee9a2a44e18f5e1a574b03fe95, State: Running, Role: LEADER
I20260927 07:05:02.657681 29581 consensus_queue.cc:237] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } }
I20260927 07:05:02.658322 29059 catalog_manager.cc:5495] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8602f2ee9a2a44e18f5e1a574b03fe95" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 } health_report { overall_health: UNKNOWN } } }
I20260927 07:05:02.680240 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.677749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:52091 (local address 127.21.2.66:37303)
0927 07:05:02.677834 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.677836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.677943 (+ 107us) server_negotiation.cc:407] Connection header received
0927 07:05:02.678012 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.678013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.678024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.678079 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.678208 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.678685 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.679496 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.679711 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.679794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.679971 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.680006 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.680074 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.680127 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.680157 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 07:05:02.680249 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.677645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37303 (local address 127.21.2.65:52091)
0927 07:05:02.677857 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.677868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.677956 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.678082 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.678083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.678085 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:05:02.678140 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.678142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.678779 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.678780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.679399 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.679401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.679794 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.679795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.679855 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.679881 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.680078 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.680080 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.680125 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.680173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260927 07:05:02.682463 29356 raft_consensus.cc:1261] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.682488 29496 raft_consensus.cc:1261] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Refusing update from remote peer 8602f2ee9a2a44e18f5e1a574b03fe95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:02.682731 29581 consensus_queue.cc:1035] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ef83a2beb84430bb771cc9fbb22224" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.682852 29659 consensus_queue.cc:1035] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7eaf9ca10748cf9c53f686700c5188" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:02.689997 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.687722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34531 (local address 127.21.2.66:45541)
0927 07:05:02.687762 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.687775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:02.687815 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:02.687930 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:02.687931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:02.687932 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:05:02.687979 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.687980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.688625 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.688626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.689153 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:02.689154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.689591 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.689592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:02.689650 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.689657 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.689826 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.689828 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:05:02.689877 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:05:02.689925 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 07:05:02.689997 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.687758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:45541 (local address 127.21.2.65:34531)
0927 07:05:02.687797 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:02.687799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:02.687807 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:02.687867 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:02.687868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:02.687877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:02.687934 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:02.688044 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.688555 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.689239 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:02.689514 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:02.689595 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:02.689719 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:02.689755 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:02.689802 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:02.689899 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:05:02.689924 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:05:02.695935 29059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.696046 29059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:02.696635 29059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39445fd5746f48b696ed4cd2b5a84a00: Sending DeleteTablet for 3 replicas of tablet e0479ade29bc4525891d693d20da6f6d
I20260927 07:05:02.696870 29196 tablet_service.cc:1514] Processing DeleteTablet for tablet e0479ade29bc4525891d693d20da6f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260927 07:05:02.696918 29476 tablet_service.cc:1514] Processing DeleteTablet for tablet e0479ade29bc4525891d693d20da6f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50074
I20260927 07:05:02.696967 29641 tablet_replica.cc:330] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: stopping tablet replica
I20260927 07:05:02.697000 29642 tablet_replica.cc:330] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: stopping tablet replica
I20260927 07:05:02.697028 29641 raft_consensus.cc:2229] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:05:02.697057 29642 raft_consensus.cc:2229] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.697079 29641 raft_consensus.cc:2258] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.697090 29642 raft_consensus.cc:2258] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.697176 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet e0479ade29bc4525891d693d20da6f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:02 UTC) from {username='oryx', principal='oryx/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50798
I20260927 07:05:02.697255 29643 tablet_replica.cc:330] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: stopping tablet replica
I20260927 07:05:02.697309 29643 raft_consensus.cc:2229] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:02.697350 29643 raft_consensus.cc:2258] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:02.697482 29642 ts_tablet_manager.cc:1905] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.697587 29641 ts_tablet_manager.cc:1905] T e0479ade29bc4525891d693d20da6f6d P 8602f2ee9a2a44e18f5e1a574b03fe95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.697664 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29131
I20260927 07:05:02.698805 29643 ts_tablet_manager.cc:1905] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:02.699178 29642 ts_tablet_manager.cc:1918] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:05:02.699237 29642 log.cc:1196] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/e0479ade29bc4525891d693d20da6f6d
I20260927 07:05:02.699502 29642 ts_tablet_manager.cc:1939] T e0479ade29bc4525891d693d20da6f6d P ce7eaf9ca10748cf9c53f686700c5188: Deleting consensus metadata
I20260927 07:05:02.699956 29046 catalog_manager.cc:4841] TS ce7eaf9ca10748cf9c53f686700c5188 (127.21.2.67:35303): tablet e0479ade29bc4525891d693d20da6f6d (table test-table [id=ce2c9123eb094543b5363209f88ea890]) successfully deleted
I20260927 07:05:02.700495 29643 ts_tablet_manager.cc:1918] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:05:02.700590 29643 log.cc:1196] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/e0479ade29bc4525891d693d20da6f6d
I20260927 07:05:02.700788 29643 ts_tablet_manager.cc:1939] T e0479ade29bc4525891d693d20da6f6d P 71ef83a2beb84430bb771cc9fbb22224: Deleting consensus metadata
I20260927 07:05:02.701124 29046 catalog_manager.cc:4841] TS 71ef83a2beb84430bb771cc9fbb22224 (127.21.2.66:37303): tablet e0479ade29bc4525891d693d20da6f6d (table test-table [id=ce2c9123eb094543b5363209f88ea890]) successfully deleted
W20260927 07:05:02.704408 29044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:34531 (error 108)
I20260927 07:05:02.704679 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29271
I20260927 07:05:02.705418 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:02.704761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:34531 (local address 127.0.0.1:34456)
0927 07:05:02.705318 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:02.705367 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":42,"threads_started":1}
W20260927 07:05:02.705648 29044 catalog_manager.cc:4584] TS 8602f2ee9a2a44e18f5e1a574b03fe95 (127.21.2.65:34531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0479ade29bc4525891d693d20da6f6d: Network error: Client connection negotiation failed: client connection to 127.21.2.65:34531: connect: Connection refused (error 111)
I20260927 07:05:02.711575 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29411
I20260927 07:05:02.717721 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29027
2026-09-27T07:05:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2873 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29678](info): set up 2 sockets
Sep 27 07:05:02 dist-test-slave-ltqz krb5kdc[29678](info): commencing operation
krb5kdc: starting...
W20260927 07:05:03.757870 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:05:03 dist-test-slave-ltqz krb5kdc[29678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:05:03.882296 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46513
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:03.965667 29691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:03.965884 29691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:03.965907 29691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:03.967244 29691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:03.967296 29691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:03.967311 29691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:03.967324 29691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:03.967337 29691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:03.969123 29691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:46513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:03.969408 29691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:03.969673 29691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:03.974551 29699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:03.974607 29696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:03.974618 29697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:03.974817 29691 server_base.cc:1034] running on GCE node
I20260927 07:05:03.975096 29691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:03.975446 29691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:03.976600 29691 hybrid_clock.cc:648] HybridClock initialized: now 1790492703976578 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T07:05:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1261 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:05:04 dist-test-slave-ltqz krb5kdc[29704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:05:04 dist-test-slave-ltqz krb5kdc[29704](info): set up 2 sockets
Sep 27 07:05:04 dist-test-slave-ltqz krb5kdc[29704](info): commencing operation
krb5kdc: starting...
W20260927 07:05:05.019963 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:05:05.137878 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40019
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:05.219647 29717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:05.219880 29717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:05.219920 29717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:05.221073 29717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:05.221143 29717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:05.221169 29717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:05.221197 29717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:05.221225 29717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:05.222764 29717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:05.223042 29717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:05.223263 29717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:05.228148 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.228121 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.228155 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.228261 29717 server_base.cc:1034] running on GCE node
I20260927 07:05:05.228591 29717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:05.228879 29717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:05.230017 29717 hybrid_clock.cc:648] HybridClock initialized: now 1790492705230007 us; error 32 us; skew 500 ppm
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:05.235152 29717 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:05:05.235639 29717 webserver.cc:460] Webserver started at http://127.21.2.126:38791/ using document root <none> and password file <none>
I20260927 07:05:05.235834 29717 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:05.235877 29717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:05.235963 29717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:05.236946 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "ab96565094a64dee9f893177ae23e07b"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "4b9aaceed2854dbb76000c19fbc3c94f"
server_key_iv: "9d13c645868652447109ca32c2221b10"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.237258 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "ab96565094a64dee9f893177ae23e07b"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "4b9aaceed2854dbb76000c19fbc3c94f"
server_key_iv: "9d13c645868652447109ca32c2221b10"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.238495 29717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:05.239180 29732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.239404 29717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:05.239512 29717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "ab96565094a64dee9f893177ae23e07b"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "4b9aaceed2854dbb76000c19fbc3c94f"
server_key_iv: "9d13c645868652447109ca32c2221b10"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.239599 29717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:05.262728 29717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:05.264923 29717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:05.265096 29717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:05.269229 29717 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40019
I20260927 07:05:05.269266 29784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40019 every 8 connection(s)
I20260927 07:05:05.269613 29717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:05.270459 29785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:05.272624 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29717
I20260927 07:05:05.272744 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:05:05.272940 21513 external_mini_cluster.cc:1428] Setting key 61b086c4f8af67915c2a2633d1e9e365
I20260927 07:05:05.273036 29785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Bootstrap starting.
I20260927 07:05:05.273737 29785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:05.274039 29785 log.cc:824] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:05.274864 29785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: No bootstrap required, opened a new log
I20260927 07:05:05.277130 29785 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:05.277304 29785 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:05.277348 29785 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab96565094a64dee9f893177ae23e07b, State: Initialized, Role: FOLLOWER
I20260927 07:05:05.277480 29785 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:05.277562 29785 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:05.277607 29785 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:05.277654 29785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:05.278319 29785 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:05.278409 29785 leader_election.cc:302] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab96565094a64dee9f893177ae23e07b; no voters:
I20260927 07:05:05.278601 29785 leader_election.cc:288] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:05:05.278721 29790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:05.278847 29785 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:05.278956 29790 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 LEADER]: Becoming Leader. State: Replica: ab96565094a64dee9f893177ae23e07b, State: Running, Role: LEADER
I20260927 07:05:05.279059 29790 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:05.279531 29792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab96565094a64dee9f893177ae23e07b. Latest consensus state: current_term: 1 leader_uuid: "ab96565094a64dee9f893177ae23e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } } }
I20260927 07:05:05.279536 29791 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab96565094a64dee9f893177ae23e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } } }
I20260927 07:05:05.279841 29792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:05.279855 29791 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:05.280102 29800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:05.280723 29800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:05.282439 29800 catalog_manager.cc:1347] Generated new cluster ID: 7523f96c3f5447d788317a743825fa7f
I20260927 07:05:05.282497 29800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:05.285008 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.273504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.21.2.126:40019)
0927 07:05:05.273683 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:05.273691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:05.273721 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:05:05.274221 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:05.274244 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:05.274312 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:05.274470 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:05.275309 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.276219 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.277347 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.277923 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.278005 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.281070 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:05.281075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:05.281076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:05.281088 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:05.283283 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.283734 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.283737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.283738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.283838 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.284063 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.284064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.284064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.284351 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:05.284490 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:05.284676 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 07:05:05.284805 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260927 07:05:05.286949 29800 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:05:05.287691 29800 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:05.298044 29800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Generated new TSK 0
I20260927 07:05:05.298276 29800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:05:05.324366 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:05.405315 29811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:05.405555 29811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:05.405594 29811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:05.406803 29811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:05.406874 29811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:05.406922 29811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:05.408471 29811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:05.408753 29811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:05.408990 29811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:05.413843 29816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.413972 29817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.413846 29819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.414242 29811 server_base.cc:1034] running on GCE node
I20260927 07:05:05.414456 29811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:05.414731 29811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:05.415905 29811 hybrid_clock.cc:648] HybridClock initialized: now 1790492705415889 us; error 24 us; skew 500 ppm
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:05.418845 29811 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:05:05.419339 29811 webserver.cc:460] Webserver started at http://127.21.2.65:41761/ using document root <none> and password file <none>
I20260927 07:05:05.419616 29811 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:05.419685 29811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:05.419847 29811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:05.420894 29811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "44f98ef34faa405f98334a44acf17ada"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b91462e7cafdf35871bc17ed1899aa03"
server_key_iv: "f668624c42338d27682e18c558bdb5cb"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.421270 29811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "44f98ef34faa405f98334a44acf17ada"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b91462e7cafdf35871bc17ed1899aa03"
server_key_iv: "f668624c42338d27682e18c558bdb5cb"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.422573 29811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:05.423287 29826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.423458 29811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:05.423560 29811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "44f98ef34faa405f98334a44acf17ada"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b91462e7cafdf35871bc17ed1899aa03"
server_key_iv: "f668624c42338d27682e18c558bdb5cb"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.423638 29811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:05.433431 29811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:05.435222 29811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:05.435433 29811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:05.435669 29811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:05.435997 29811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:05.436053 29811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.436089 29811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:05.436115 29811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.442129 29811 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:41111
I20260927 07:05:05.442195 29939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:41111 every 8 connection(s)
I20260927 07:05:05.442553 29811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:05.449673 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29811
I20260927 07:05:05.449786 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:05:05.449975 21513 external_mini_cluster.cc:1428] Setting key 933e48cde0d7d9725b963dc732b38029
I20260927 07:05:05.452414 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.443425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:56787 (local address 127.21.2.126:40019)
0927 07:05:05.443478 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:05.443480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:05.443715 (+ 235us) server_negotiation.cc:407] Connection header received
0927 07:05:05.444211 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:05.444212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:05.444225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:05.444288 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:05.445204 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.445790 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.446771 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.446992 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.447073 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.449222 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:05.449223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:05.449224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:05.449231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:05.450801 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.451278 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.451279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.451280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.451330 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.451647 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.451647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.451648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.451759 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:05.451820 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:05.452177 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 07:05:05.452282 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:05:05.452574 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.443368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:56787)
0927 07:05:05.443556 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:05.443598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:05.444077 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:05.444339 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:05.444342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:05.444896 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:05.445087 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.445092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.445882 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.445883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.446673 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.446675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.447118 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.447119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.447241 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.447757 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:05.447764 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:05.449134 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:05.450876 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.450883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.450886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.451198 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.451431 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.451432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.451433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.451578 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.451837 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:05.451842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:05.452016 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 07:05:05.452133 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 07:05:05.452321 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 07:05:05.453158 29940 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40019
I20260927 07:05:05.453279 29940 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:05.453459 29940 heartbeater.cc:511] Master 127.21.2.126:40019 requested a full tablet report, sending...
I20260927 07:05:05.454011 29749 ts_manager.cc:194] Registered new tserver with Master: 44f98ef34faa405f98334a44acf17ada (127.21.2.65:41111)
I20260927 07:05:05.454636 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:56787
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:05:05.474124 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:05.558213 29945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:05.558547 29945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:05.558576 29945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:05.559940 29945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:05.560002 29945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:05.560034 29945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:05.561592 29945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:05.561862 29945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:05.562124 29945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:05:05.567132 29945 server_base.cc:1034] running on GCE node
W20260927 07:05:05.567065 29953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.567063 29951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.567075 29950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.567520 29945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:05.567798 29945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:05.568940 29945 hybrid_clock.cc:648] HybridClock initialized: now 1790492705568918 us; error 39 us; skew 500 ppm
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:05.571491 29945 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:05:05.571904 29945 webserver.cc:460] Webserver started at http://127.21.2.66:37605/ using document root <none> and password file <none>
I20260927 07:05:05.572099 29945 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:05.572139 29945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:05.572228 29945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:05.573266 29945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "8e1549211e7842c9bbe042c0fcd3d884"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b27a82b8a8423d120d1c3e6436f703a8"
server_key_iv: "b26df77613352bf20da93d30764ea726"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.573567 29945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "8e1549211e7842c9bbe042c0fcd3d884"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b27a82b8a8423d120d1c3e6436f703a8"
server_key_iv: "b26df77613352bf20da93d30764ea726"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.575395 29945 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 07:05:05.576164 29960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.576360 29945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:05.576476 29945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "8e1549211e7842c9bbe042c0fcd3d884"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b27a82b8a8423d120d1c3e6436f703a8"
server_key_iv: "b26df77613352bf20da93d30764ea726"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.576556 29945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:05.584319 29945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:05.586644 29945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:05.586808 29945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:05.587036 29945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:05.587427 29945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:05.587487 29945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.587529 29945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:05.587572 29945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.593438 29945 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:35847
I20260927 07:05:05.593478 30073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:35847 every 8 connection(s)
I20260927 07:05:05.593839 29945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:05:05.599189 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 29945
I20260927 07:05:05.599326 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:05:05.599551 21513 external_mini_cluster.cc:1428] Setting key 9850a892826817382736144e1cdd2982
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:05.604130 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.594771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:52533 (local address 127.21.2.126:40019)
0927 07:05:05.594834 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:05.594836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:05.594978 (+ 142us) server_negotiation.cc:407] Connection header received
0927 07:05:05.595540 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:05.595542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:05.595553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:05.595620 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:05.596579 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.597086 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.598142 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.598351 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.598412 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.600990 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:05.600992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:05.600993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:05.601002 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:05.602594 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.603034 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.603036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.603036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.603100 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.603359 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.603425 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.603425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.603544 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:05.603628 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:05.603945 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 07:05:05.604031 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:05:05.604365 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.594681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:52533)
0927 07:05:05.594849 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:05.594891 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:05.595418 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:05.595697 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:05.595701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:05.596288 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:05.596474 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.596478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.597256 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.597257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.598051 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.598053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.598434 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.598435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.598534 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.599120 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:05.599132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:05.600855 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:05.602706 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.602714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.602716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.602951 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.603171 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.603172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.603172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.603289 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.603618 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:05.603625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:05.603805 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 07:05:05.603921 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 07:05:05.604104 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260927 07:05:05.605264 30074 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40019
I20260927 07:05:05.605381 30074 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:05.605566 30074 heartbeater.cc:511] Master 127.21.2.126:40019 requested a full tablet report, sending...
I20260927 07:05:05.606041 29749 ts_manager.cc:194] Registered new tserver with Master: 8e1549211e7842c9bbe042c0fcd3d884 (127.21.2.66:35847)
I20260927 07:05:05.606604 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:52533
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:05:05.624420 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:05.705809 30079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:05.706070 30079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:05.706111 30079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:05.707424 30079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:05.707507 30079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:05.707556 30079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:05.709069 30079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:05.709360 30079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:05.709601 30079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:05.714869 30087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.715024 30079 server_base.cc:1034] running on GCE node
W20260927 07:05:05.714881 30085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.714946 30084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.715440 30079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:05.715721 30079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:05.716892 30079 hybrid_clock.cc:648] HybridClock initialized: now 1790492705716867 us; error 43 us; skew 500 ppm
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:05.719594 30079 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:05:05.720003 30079 webserver.cc:460] Webserver started at http://127.21.2.67:36335/ using document root <none> and password file <none>
I20260927 07:05:05.720191 30079 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:05.720239 30079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:05.720358 30079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:05.721385 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "6daa85e2330a4f32ab14c86a01ec7bb5"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "2eaf9b90d752dc1fd11111c6e517e54e"
server_key_iv: "858b7bc0478fff20f5571f04d5f375e9"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.721701 30079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "6daa85e2330a4f32ab14c86a01ec7bb5"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "2eaf9b90d752dc1fd11111c6e517e54e"
server_key_iv: "858b7bc0478fff20f5571f04d5f375e9"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.723014 30079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 07:05:05.723829 30094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.724010 30079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:05:05.724112 30079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "6daa85e2330a4f32ab14c86a01ec7bb5"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "2eaf9b90d752dc1fd11111c6e517e54e"
server_key_iv: "858b7bc0478fff20f5571f04d5f375e9"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.724175 30079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:05.741707 30079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:05.743903 30079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:05.744021 30079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:05.744205 30079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:05.744530 30079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:05.744588 30079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.744634 30079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:05.744678 30079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.750787 30079 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:38111
I20260927 07:05:05.750875 30207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:38111 every 8 connection(s)
I20260927 07:05:05.751343 30079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:05.759724 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30079
I20260927 07:05:05.759850 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:05:05.760041 21513 external_mini_cluster.cc:1428] Setting key 0485b1bafd78f635fb3b3beccf3dcf64
I20260927 07:05:05.762183 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.752469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:43185 (local address 127.21.2.126:40019)
0927 07:05:05.752540 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:05.752542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:05.752785 (+ 243us) server_negotiation.cc:407] Connection header received
0927 07:05:05.753345 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:05.753346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:05.753360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:05.753419 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:05.754388 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.754912 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.755967 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.756352 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.756547 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.758885 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:05.758887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:05.758887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:05.758897 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:05.760509 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.761045 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.761046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.761046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.761114 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:05.761418 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:05.761419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:05.761419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:05.761583 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:05.761669 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:05.761991 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 07:05:05.762071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:05:05.762415 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:05.752455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:43185)
0927 07:05:05.752653 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:05.752693 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:05.753234 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:05.753495 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:05.753498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:05.754078 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:05.754283 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.754288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.755016 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.755018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.755868 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:05.755871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:05.756560 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:05.756562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:05.756705 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:05.757248 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:05.757257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:05.758776 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:05.760624 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.760633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.760636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.760970 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.761214 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:05.761215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:05.761215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:05.761361 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:05.761675 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:05.761681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:05.761862 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 07:05:05.761972 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:05:05.762140 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260927 07:05:05.762980 30208 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40019
I20260927 07:05:05.763085 30208 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:05.763267 30208 heartbeater.cc:511] Master 127.21.2.126:40019 requested a full tablet report, sending...
I20260927 07:05:05.763674 29749 ts_manager.cc:194] Registered new tserver with Master: 6daa85e2330a4f32ab14c86a01ec7bb5 (127.21.2.67:38111)
I20260927 07:05:05.764051 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:43185
I20260927 07:05:05.772526 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
Entry for principal oryx/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab.
WARNING: no policy specified for oryx/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
Entry for principal oryx/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab.
WARNING: no policy specified for oryx/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
Entry for principal oryx/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab.
I20260927 07:05:05.843915 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29811
I20260927 07:05:05.848453 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29945
I20260927 07:05:05.855218 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30079
I20260927 07:05:05.859562 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 29717
I20260927 07:05:05.864813 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40019
--webserver_interface=127.21.2.126
--webserver_port=38791
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:05.945937 30217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:05.946164 30217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:05.946189 30217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:05.947335 30217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:05.947393 30217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:05.947410 30217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:05.947423 30217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:05.947435 30217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:05.948868 30217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:05.949122 30217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:05.949385 30217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:05.954598 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.954545 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:05.954546 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:05.954870 30217 server_base.cc:1034] running on GCE node
I20260927 07:05:05.955099 30217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:05.955451 30217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:05.956611 30217 hybrid_clock.cc:648] HybridClock initialized: now 1790492705956597 us; error 30 us; skew 500 ppm
Sep 27 07:05:05 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492705, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:05.959357 30217 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:05:05.959828 30217 webserver.cc:460] Webserver started at http://127.21.2.126:38791/ using document root <none> and password file <none>
I20260927 07:05:05.960012 30217 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:05.960063 30217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:05.961378 30217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:05.961997 30232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:05.962179 30217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:05.962289 30217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "ab96565094a64dee9f893177ae23e07b"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "4b9aaceed2854dbb76000c19fbc3c94f"
server_key_iv: "9d13c645868652447109ca32c2221b10"
server_key_version: "encryptionkey@0"
I20260927 07:05:05.962666 30217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:05.979637 30217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:05.981613 30217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:05.981784 30217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:05.985690 30217 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40019
I20260927 07:05:05.985728 30284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40019 every 8 connection(s)
I20260927 07:05:05.986112 30217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:05.987581 30285 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:05:05.988225 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Bootstrap starting.
I20260927 07:05:05.989717 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30217
I20260927 07:05:05.990013 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:41111
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:40019
--webserver_port=41761
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:05:05.996780 30285 log.cc:824] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:05.998449 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:05:05.998769 30285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Bootstrap complete.
I20260927 07:05:06.001168 30285 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:06.001411 30285 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab96565094a64dee9f893177ae23e07b, State: Initialized, Role: FOLLOWER
I20260927 07:05:06.001576 30285 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:06.001648 30285 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:06.001698 30285 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:06.001746 30285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:05:06.002858 30285 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:06.002974 30285 leader_election.cc:302] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab96565094a64dee9f893177ae23e07b; no voters:
I20260927 07:05:06.003182 30285 leader_election.cc:288] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:05:06.003217 30289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:05:06.003331 30289 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [term 2 LEADER]: Becoming Leader. State: Replica: ab96565094a64dee9f893177ae23e07b, State: Running, Role: LEADER
I20260927 07:05:06.003449 30289 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } }
I20260927 07:05:06.003477 30285 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:06.004292 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab96565094a64dee9f893177ae23e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } } }
I20260927 07:05:06.004379 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:06.004779 30299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:05:06.004882 30291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab96565094a64dee9f893177ae23e07b. Latest consensus state: current_term: 2 leader_uuid: "ab96565094a64dee9f893177ae23e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab96565094a64dee9f893177ae23e07b" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40019 } } }
I20260927 07:05:06.004959 30291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:06.005283 30299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:06.005410 30299 catalog_manager.cc:1259] Loaded cluster ID: 7523f96c3f5447d788317a743825fa7f
I20260927 07:05:06.005447 30299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:06.006548 30299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:06.006717 30299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab96565094a64dee9f893177ae23e07b: Loaded TSK: 0
I20260927 07:05:06.006937 30299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:05:06.077770 30287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:06.078009 30287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:06.078050 30287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:06.079241 30287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:06.079306 30287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:06.079352 30287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:06.080835 30287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.65.keytab
--principal=oryx/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:41111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=41761
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:06.081073 30287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:06.081313 30287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:06.087105 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:06.087162 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:06.087138 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:06.087337 30287 server_base.cc:1034] running on GCE node
I20260927 07:05:06.087545 30287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:06.087810 30287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:06.088966 30287 hybrid_clock.cc:648] HybridClock initialized: now 1790492706088946 us; error 36 us; skew 500 ppm
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492706, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:06.091663 30287 init.cc:377] Logged in from keytab as oryx/127.21.2.65@KRBTEST.COM (short username oryx)
I20260927 07:05:06.092262 30287 webserver.cc:460] Webserver started at http://127.21.2.65:41761/ using document root <none> and password file <none>
I20260927 07:05:06.092504 30287 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:06.092590 30287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:06.093967 30287 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:05:06.094624 30321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.094882 30287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:06.094983 30287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "44f98ef34faa405f98334a44acf17ada"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b91462e7cafdf35871bc17ed1899aa03"
server_key_iv: "f668624c42338d27682e18c558bdb5cb"
server_key_version: "encryptionkey@0"
I20260927 07:05:06.095479 30287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:06.122581 30287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:06.125026 30287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:06.125164 30287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:06.125409 30287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:06.125744 30287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:06.125784 30287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.125814 30287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:06.125831 30287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.132993 30287 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:41111
I20260927 07:05:06.133034 30434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:41111 every 8 connection(s)
I20260927 07:05:06.133456 30287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:05:06.135898 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30287
I20260927 07:05:06.136202 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:35847
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:40019
--webserver_port=37605
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.144426 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.134328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:57583)
0927 07:05:06.134551 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.134599 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.135212 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.135758 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.135762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.136644 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.136974 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.136982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.138256 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.138257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.139523 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.139526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.140313 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.140315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.140474 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.141112 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.141123 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.144175 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":122,"threads_started":1}
W20260927 07:05:06.144685 30435 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40019 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.2.126:40019: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
W20260927 07:05:06.144824 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.134461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:57583 (local address 127.21.2.126:40019)
0927 07:05:06.134648 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.134661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.134690 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:05:06.135514 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.135520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.135564 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.135731 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.137123 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.138123 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.139608 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.140203 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.140316 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.144348 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:57583: BlockingRecv error: recv got EOF from 127.21.2.65:57583 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.150228 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.144932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:59799)
0927 07:05:06.145186 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.145199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.145255 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.145427 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.145428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.145597 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.145659 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.145661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.146565 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.146567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.147513 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.147516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.147852 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.147853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.147918 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.148338 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.148343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.150140 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
W20260927 07:05:06.150281 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.145115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:59799 (local address 127.21.2.126:40019)
0927 07:05:06.145176 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.145178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.145238 (+ 60us) server_negotiation.cc:407] Connection header received
0927 07:05:06.145321 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.145322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.145333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.145394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.145750 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.146471 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.147594 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.147780 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.147835 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.150213 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:59799: BlockingRecv error: recv got EOF from 127.21.2.65:59799 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.154680 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.150519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:45137)
0927 07:05:06.150620 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.150631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.150679 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.150833 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.150834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.150959 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.150983 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.150984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.151649 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.151650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.152153 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.152154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.152515 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.152516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.152567 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.152995 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.153000 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.154597 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 07:05:06.154729 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.150572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:45137 (local address 127.21.2.126:40019)
0927 07:05:06.150630 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.150632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.150664 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:05:06.150747 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.150748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.150759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.150834 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.151067 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.151572 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.152243 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.152430 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.152513 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.154666 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:45137: BlockingRecv error: recv got EOF from 127.21.2.65:45137 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 07:05:06.154824 30435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:05:06.221174 30439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:06.221407 30439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:06.221444 30439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:06.222584 30439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:06.222649 30439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:06.222694 30439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:06.224232 30439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.66.keytab
--principal=oryx/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:35847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=37605
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:06.224506 30439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:06.224731 30439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:06.229679 30445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:06.229673 30446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:06.229696 30439 server_base.cc:1034] running on GCE node
W20260927 07:05:06.229679 30448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:06.230171 30439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:06.230480 30439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:06.231637 30439 hybrid_clock.cc:648] HybridClock initialized: now 1790492706231632 us; error 36 us; skew 500 ppm
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492706, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:06.234059 30439 init.cc:377] Logged in from keytab as oryx/127.21.2.66@KRBTEST.COM (short username oryx)
I20260927 07:05:06.234529 30439 webserver.cc:460] Webserver started at http://127.21.2.66:37605/ using document root <none> and password file <none>
I20260927 07:05:06.234764 30439 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:06.234848 30439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:06.236258 30439 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:05:06.236846 30455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.237027 30439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:05:06.237124 30439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "8e1549211e7842c9bbe042c0fcd3d884"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "b27a82b8a8423d120d1c3e6436f703a8"
server_key_iv: "b26df77613352bf20da93d30764ea726"
server_key_version: "encryptionkey@0"
I20260927 07:05:06.237475 30439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:06.255729 30439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:06.258194 30439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:06.258370 30439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:06.258618 30439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:06.258930 30439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:06.258985 30439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.259029 30439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:06.259054 30439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.265204 30439 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:35847
I20260927 07:05:06.265249 30568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:35847 every 8 connection(s)
I20260927 07:05:06.265592 30439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.271898 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30439
I20260927 07:05:06.272173 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:38111
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:40019
--webserver_port=36335
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:06.272800 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.266287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:41371 (local address 127.21.2.126:40019)
0927 07:05:06.266387 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.266389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.266554 (+ 165us) server_negotiation.cc:407] Connection header received
0927 07:05:06.267035 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.267036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.267048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.267109 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.268095 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.268572 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.269519 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.269689 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.269740 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.272745 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:41371: BlockingRecv error: recv got EOF from 127.21.2.66:41371 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:05:06.272854 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.266262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:41371)
0927 07:05:06.266424 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.266465 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.266956 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.267183 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.267186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.267827 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.268014 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.268019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.268654 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.268655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.269448 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.269450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.269789 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.269790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.269896 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.270373 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.270381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.272660 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
W20260927 07:05:06.273056 30569 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40019 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.2.126:40019: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.277648 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.273309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:42799)
0927 07:05:06.273350 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.273366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.273416 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.273543 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.273544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.273668 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.273709 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.273711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.274323 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.274324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.274830 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.274832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.275198 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.275199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.275274 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.275767 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.275771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.277562 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 07:05:06.277717 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.273360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:42799 (local address 127.21.2.126:40019)
0927 07:05:06.273421 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.273423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.273449 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:05:06.273473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.273474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.273482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.273531 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.273785 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.274245 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.274912 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.275114 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.275197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.277639 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:42799: BlockingRecv error: recv got EOF from 127.21.2.66:42799 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.283340 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.278066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:36749)
0927 07:05:06.278184 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.278197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.278283 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.278514 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.278515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.278700 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.278748 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.278750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.279810 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.279811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.280276 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.280278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.280761 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.280762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.280796 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.281235 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.281239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.283266 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 07:05:06.283524 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.278132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:36749 (local address 127.21.2.126:40019)
0927 07:05:06.278176 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.278178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.278257 (+ 79us) server_negotiation.cc:407] Connection header received
0927 07:05:06.278371 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.278372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.278401 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.278464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.278847 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.279713 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.280357 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.280686 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.280765 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.283441 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:36749: BlockingRecv error: recv got EOF from 127.21.2.66:36749 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 07:05:06.283608 30569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:05:06.353896 30572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:06.354130 30572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:06.354156 30572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:06.355327 30572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:06.355396 30572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:06.355475 30572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:06.356986 30572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/krb5kdc/oryx_127.21.2.67.keytab
--principal=oryx/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:38111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=36335
--tserver_master_addrs=127.21.2.126:40019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:06.357182 30572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:06.357370 30572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:06.362401 30577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:06.362473 30580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:06.362401 30578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:06.362552 30572 server_base.cc:1034] running on GCE node
I20260927 07:05:06.362746 30572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:06.362982 30572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:06.364131 30572 hybrid_clock.cc:648] HybridClock initialized: now 1790492706364094 us; error 71 us; skew 500 ppm
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492706, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:06.367079 30572 init.cc:377] Logged in from keytab as oryx/127.21.2.67@KRBTEST.COM (short username oryx)
I20260927 07:05:06.367592 30572 webserver.cc:460] Webserver started at http://127.21.2.67:36335/ using document root <none> and password file <none>
I20260927 07:05:06.367781 30572 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:06.367833 30572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:06.369105 30572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:06.369673 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.369835 30572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:06.369925 30572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "6daa85e2330a4f32ab14c86a01ec7bb5"
format_stamp: "Formatted at 2026-09-27 07:05:05 on dist-test-slave-ltqz"
server_key: "2eaf9b90d752dc1fd11111c6e517e54e"
server_key_iv: "858b7bc0478fff20f5571f04d5f375e9"
server_key_version: "encryptionkey@0"
I20260927 07:05:06.370249 30572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:06.382895 30572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:06.385162 30572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:06.385277 30572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:06.385459 30572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:06.385697 30572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:06.385730 30572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.385751 30572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:06.385764 30572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:06.391258 30572 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:38111
I20260927 07:05:06.391330 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:38111 every 8 connection(s)
I20260927 07:05:06.391687 30572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:05:06.397198 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30572
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:05:06.399605 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.392586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:52879 (local address 127.21.2.126:40019)
0927 07:05:06.392634 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.392636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.392876 (+ 240us) server_negotiation.cc:407] Connection header received
0927 07:05:06.393393 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.393394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.393407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.393459 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.394373 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.394863 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.395933 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.396124 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.396220 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.399559 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:52879: BlockingRecv error: recv got EOF from 127.21.2.67:52879 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:05:06.399676 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.392559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:52879)
0927 07:05:06.392760 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.392793 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.393287 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.393541 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.393545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.394092 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.394275 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.394279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.394959 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.394960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.395828 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.395830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.396236 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.396237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.396352 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.396788 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.396796 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.399482 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
W20260927 07:05:06.399955 30701 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40019 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.2.126:40019: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.404898 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.400198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:58227)
0927 07:05:06.400287 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.400297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.400350 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.400487 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.400488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.400624 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.400664 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.400666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.401293 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.401294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.401795 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.401796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.402272 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.402273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.402365 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.402828 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.402838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.404822 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 07:05:06.404980 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.400245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:58227 (local address 127.21.2.126:40019)
0927 07:05:06.400302 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.400304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.400336 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:05:06.400414 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.400414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.400422 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.400465 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.400729 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.401213 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.401893 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.402168 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.402243 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.404904 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:58227: BlockingRecv error: recv got EOF from 127.21.2.67:58227 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:05:06.406165 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.397755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.21.2.126:40019)
0927 07:05:06.397865 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.397867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.397875 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:06.397953 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.397954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.397964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.398034 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.398787 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.399354 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.400520 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.400758 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.400824 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.402025 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:06.402029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:06.402029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:06.402041 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:06.404121 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:06.404573 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:06.404593 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:06.404593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:06.404725 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:06.405349 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:06.405350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:06.405350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:06.405682 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:06.405778 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:06.405961 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 07:05:06.406068 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:06 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:06.409574 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:06.405286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:38547)
0927 07:05:06.405409 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:06.405421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:06.405468 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:06.405585 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:06.405585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:06.405699 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:06.405752 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.405753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.406570 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.406571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.407281 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:06.407283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.407657 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.407657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:06.407694 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.408002 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:06.408006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:06.409532 (+ 1526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 07:05:06.409706 30701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:05:06.409726 30438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:06.405416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:38547 (local address 127.21.2.126:40019)
0927 07:05:06.405478 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:06.405480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:06.405488 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:06.405509 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:06.405509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:06.405519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:06.405581 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:06.405827 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.406484 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.407344 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:06.407595 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:06.407648 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:06.409656 (+ 2008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:38547: BlockingRecv error: recv got EOF from 127.21.2.67:38547 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:07.160138 30706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:07.155294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:49005)
0927 07:05:07.155434 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:07.155446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:07.155503 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:07.155711 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:07.155712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:07.155870 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:07.155918 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.155920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.156595 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.156596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.157147 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.157148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.157534 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.157535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.157575 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.158038 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:07.158043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:07.160064 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":119,"threads_started":1}
W20260927 07:05:07.160213 30707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:07.155412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:49005 (local address 127.21.2.126:40019)
0927 07:05:07.155532 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:07.155534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:07.155545 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:07.155574 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:07.155575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:07.155588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:07.155700 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:07.155992 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.156508 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.157258 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.157461 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.157525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.160157 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:49005: BlockingRecv error: recv got EOF from 127.21.2.65:49005 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:07.288695 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:07.283978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:49279)
0927 07:05:07.284162 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:07.284176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:07.284249 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:07.284408 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:07.284409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:07.284584 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:07.284629 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.284631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.285326 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.285327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.285892 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.285893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.286275 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.286276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.286330 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.286800 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:07.286805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:07.288587 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
W20260927 07:05:07.288733 30707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:07.284075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49279 (local address 127.21.2.126:40019)
0927 07:05:07.284147 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:07.284149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:07.284245 (+ 96us) server_negotiation.cc:407] Connection header received
0927 07:05:07.284323 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:07.284323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:07.284341 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:07.284402 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:07.284712 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.285213 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.285986 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.286186 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.286274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.288668 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:49279: BlockingRecv error: recv got EOF from 127.21.2.66:49279 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:07 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:07.414741 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:07.410176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:37797)
0927 07:05:07.410320 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:07.410332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:07.410408 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:07.410595 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:07.410595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:07.410808 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:07.410869 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.410872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.411603 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.411604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.412150 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:07.412152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.412463 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.412464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:07.412505 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.413003 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:07.413015 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:07.414673 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260927 07:05:07.414847 30707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:07.410261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:37797 (local address 127.21.2.126:40019)
0927 07:05:07.410325 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:07.410327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:07.410407 (+ 80us) server_negotiation.cc:407] Connection header received
0927 07:05:07.410489 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:07.410490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:07.410505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:07.410560 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:07.410952 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.411535 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.412221 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:07.412402 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:07.412451 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:07.414770 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:37797: BlockingRecv error: recv got EOF from 127.21.2.67:37797 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:08.165645 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:08.160699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:41333)
0927 07:05:08.160851 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:08.160862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:08.160912 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:08.161149 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:08.161150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:08.161324 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:08.161372 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.161374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.162048 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.162049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.162683 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.162686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.163078 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.163079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.163135 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.163678 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:08.163684 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:08.165550 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
W20260927 07:05:08.165702 30711 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:08.160887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:41333 (local address 127.21.2.126:40019)
0927 07:05:08.161019 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:08.161021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:08.161029 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:08.161051 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:08.161052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:08.161062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:08.161135 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:08.161464 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.161976 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.162783 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.162980 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.163077 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.165629 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:41333: BlockingRecv error: recv got EOF from 127.21.2.65:41333 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:08.294162 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:08.289306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:49499)
0927 07:05:08.289498 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:08.289510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:08.289602 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:08.289769 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:08.289770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:08.289950 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:08.289996 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.289998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.290674 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.290675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.291205 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.291207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.291635 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.291636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.291694 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.292144 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:08.292150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:08.294067 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
W20260927 07:05:08.294211 30711 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:08.289414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49499 (local address 127.21.2.126:40019)
0927 07:05:08.289494 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:08.289496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:08.289583 (+ 87us) server_negotiation.cc:407] Connection header received
0927 07:05:08.289676 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:08.289677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:08.289696 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:08.289761 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:08.290080 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.290582 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.291294 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.291549 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.291639 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.294145 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:49499: BlockingRecv error: recv got EOF from 127.21.2.66:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:08 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:08.419926 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:08.415231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:54645)
0927 07:05:08.415410 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:08.415434 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:08.415501 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:08.415640 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:08.415641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:08.415819 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:08.415872 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.415874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.416505 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.416506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.417036 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:08.417038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.417385 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.417387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:08.417437 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.417982 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:08.417986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:08.419819 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
W20260927 07:05:08.419966 30711 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:08.415408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:54645 (local address 127.21.2.126:40019)
0927 07:05:08.415456 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:08.415458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:08.415481 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:05:08.415555 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:08.415556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:08.415568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:08.415614 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:08.415937 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.416425 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.417110 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:08.417301 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:08.417351 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:08.419900 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:54645: BlockingRecv error: recv got EOF from 127.21.2.67:54645 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:09.170850 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:09.166137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:59669)
0927 07:05:09.166297 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:09.166312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:09.166372 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:09.166507 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:09.166508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:09.166693 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:09.166745 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.166747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.167448 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.167449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.167997 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.167999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.168499 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.168500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.168546 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.168972 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:09.168977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:09.170783 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
W20260927 07:05:09.170987 30715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:09.166220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:59669 (local address 127.21.2.126:40019)
0927 07:05:09.166374 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:09.166375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:09.166403 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:05:09.166426 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:09.166427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:09.166436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:09.166486 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:09.166816 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.167310 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.168088 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.168435 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.168489 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.170878 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:59669: BlockingRecv error: recv got EOF from 127.21.2.65:59669 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:09.299633 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:09.294717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:41849)
0927 07:05:09.294901 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:09.294915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:09.294995 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:09.295175 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:09.295176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:09.295396 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:09.295445 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.295447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.296130 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.296132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.296676 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.296678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.297052 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.297054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.297120 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.297559 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:09.297564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:09.299525 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
W20260927 07:05:09.299706 30715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:09.294842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:41849 (local address 127.21.2.126:40019)
0927 07:05:09.294932 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:09.294935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:09.294974 (+ 39us) server_negotiation.cc:407] Connection header received
0927 07:05:09.295075 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:09.295076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:09.295096 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:09.295157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:09.295528 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.296034 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.296769 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.296965 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.297042 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.299658 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:41849: BlockingRecv error: recv got EOF from 127.21.2.66:41849 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:09 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:09.425741 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:09.420504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:54913)
0927 07:05:09.420802 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:09.420815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:09.420924 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:09.421090 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:09.421091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:09.421275 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:09.421322 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.421324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.421998 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.421999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.422539 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:09.422541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.422898 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.422899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:09.422966 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.423526 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:09.423539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:09.425620 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":188,"threads_started":1}
W20260927 07:05:09.425781 30715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:09.420713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:54913 (local address 127.21.2.126:40019)
0927 07:05:09.420802 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:09.420804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:09.420903 (+ 99us) server_negotiation.cc:407] Connection header received
0927 07:05:09.420991 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:09.420992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:09.421016 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:09.421084 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:09.421409 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.421901 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.422623 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:09.422817 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:09.422897 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:09.425714 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:54913: BlockingRecv error: recv got EOF from 127.21.2.67:54913 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:10.176311 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:10.171397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:54679)
0927 07:05:10.171579 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:10.171590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:10.171658 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:10.171797 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:10.171798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:10.171968 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:10.172016 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.172017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.172715 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.172716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.173356 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.173359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.173705 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.173706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.173745 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.174251 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:10.174256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:10.176252 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
W20260927 07:05:10.176461 30719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:10.171499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54679 (local address 127.21.2.126:40019)
0927 07:05:10.171607 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:10.171609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:10.171691 (+ 82us) server_negotiation.cc:407] Connection header received
0927 07:05:10.171715 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:10.171716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:10.171727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:10.171777 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:10.172083 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.172634 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.173441 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.173636 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.173693 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.176374 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:54679: BlockingRecv error: recv got EOF from 127.21.2.65:54679 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:10.308301 30720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:10.300208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:56699)
0927 07:05:10.300382 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:10.300395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:10.300490 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:10.300651 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:10.300652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:10.300844 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:10.300893 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.300895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.301591 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.301592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.302138 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.302140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.302523 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.302524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.302586 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.306222 (+ 3636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:10.306228 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:10.308243 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
W20260927 07:05:10.308506 30719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:10.300288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:56699 (local address 127.21.2.126:40019)
0927 07:05:10.300384 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:10.300386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:10.300470 (+ 84us) server_negotiation.cc:407] Connection header received
0927 07:05:10.300557 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:10.300558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:10.300570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:10.300644 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:10.300983 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.301497 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.302225 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.302433 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.302521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.308427 (+ 5906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:56699: BlockingRecv error: recv got EOF from 127.21.2.66:56699 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:10 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:10.430977 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:10.426284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:54101)
0927 07:05:10.426419 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:10.426431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:10.426514 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:10.426673 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:10.426674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:10.426869 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:10.426917 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.426919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.427638 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.427640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.428177 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:10.428179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.428562 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.428562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:10.428621 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.429078 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:10.429082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:10.430884 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260927 07:05:10.431018 30719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:10.426469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:54101 (local address 127.21.2.126:40019)
0927 07:05:10.426546 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:10.426548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:10.426557 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:10.426581 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:10.426582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:10.426601 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:10.426665 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:10.427001 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.427557 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.428267 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:10.428457 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:10.428548 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:10.430957 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:54101: BlockingRecv error: recv got EOF from 127.21.2.67:54101 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:11.181470 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:11.176841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:34205)
0927 07:05:11.177006 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:11.177016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:11.177069 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:11.177246 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:11.177247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:11.177417 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:11.177469 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.177471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.178114 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.178115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.178638 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.178640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.179060 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.179061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.179112 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.179703 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:11.179707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:11.181412 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
W20260927 07:05:11.181643 30723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:11.176894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:34205 (local address 127.21.2.126:40019)
0927 07:05:11.177121 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:11.177123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:11.177131 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:11.177151 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:11.177152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:11.177162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:11.177224 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:11.177542 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.178043 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.178758 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.178959 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.179034 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.181555 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:34205: BlockingRecv error: recv got EOF from 127.21.2.65:34205 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":173,"threads_started":1}
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:11.313644 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:11.308835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:54817)
0927 07:05:11.308970 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:11.308985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:11.309045 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:11.309206 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:11.309207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:11.309458 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:11.309518 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.309520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.310163 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.310164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.310977 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.310979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.311301 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.311302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.311339 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.311857 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:11.311869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:11.313588 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260927 07:05:11.313781 30723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:11.308960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:54817 (local address 127.21.2.126:40019)
0927 07:05:11.309018 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:11.309019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:11.309027 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:11.309123 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:11.309124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:11.309137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:11.309199 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:11.309595 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.310081 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.311048 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.311233 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.311289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.313699 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:54817: BlockingRecv error: recv got EOF from 127.21.2.66:54817 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:11 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:11.436244 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:11.431575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:60017)
0927 07:05:11.431730 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:11.431745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:11.431827 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:11.431995 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:11.431996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:11.432170 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:11.432219 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.432221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.432893 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.432894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.433422 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:11.433424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.433781 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.433782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:11.433852 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.434295 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:11.434299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:11.436147 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260927 07:05:11.436295 30723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:11.431649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:60017 (local address 127.21.2.126:40019)
0927 07:05:11.431710 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:11.431712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:11.431817 (+ 105us) server_negotiation.cc:407] Connection header received
0927 07:05:11.431885 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:11.431886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:11.431902 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:11.431986 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:11.432303 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.432796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.433511 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:11.433705 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:11.433783 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:11.436227 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:60017: BlockingRecv error: recv got EOF from 127.21.2.67:60017 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:12.186842 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:12.181996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:52233)
0927 07:05:12.182169 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:12.182209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:12.182266 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:12.182430 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:12.182431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:12.182603 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:12.182652 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.182653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.183285 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.183286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.183830 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.183831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.184181 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.184182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.184226 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.184814 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:12.184819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:12.186747 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":121,"threads_started":1}
W20260927 07:05:12.186962 30727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:12.182150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:52233 (local address 127.21.2.126:40019)
0927 07:05:12.182298 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:12.182301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:12.182309 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:12.182332 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:12.182332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:12.182342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:12.182413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:12.182724 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.183214 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.183908 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.184094 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.184146 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.186856 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:52233: BlockingRecv error: recv got EOF from 127.21.2.65:52233 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:12.318825 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:12.314093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:54601)
0927 07:05:12.314265 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:12.314278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:12.314371 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:12.314525 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:12.314526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:12.314715 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:12.314757 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.314759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.315507 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.315508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.316054 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.316055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.316456 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.316457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.316518 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.316966 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:12.316971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:12.318769 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
W20260927 07:05:12.318954 30727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:12.314178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:54601 (local address 127.21.2.126:40019)
0927 07:05:12.314255 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:12.314257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:12.314352 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:05:12.314433 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:12.314434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:12.314454 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:12.314520 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:12.314838 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.315335 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.316148 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.316343 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.316455 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.318875 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:54601: BlockingRecv error: recv got EOF from 127.21.2.66:54601 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:12 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:12.441208 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:12.436839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:38011)
0927 07:05:12.436973 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:12.436983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:12.437036 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:12.437142 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:12.437143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:12.437291 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:12.437338 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.437339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.437967 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.437968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.438486 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:12.438488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.438814 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.438814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:12.438854 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.439317 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:12.439321 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:12.441152 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260927 07:05:12.441380 30727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:12.436914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:38011 (local address 127.21.2.126:40019)
0927 07:05:12.436977 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:12.436978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:12.437021 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:05:12.437077 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:12.437077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:12.437088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:12.437137 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:12.437399 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.437883 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.438559 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:12.438746 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:12.438802 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:12.441272 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:38011: BlockingRecv error: recv got EOF from 127.21.2.67:38011 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:13.191713 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:13.187317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:34261)
0927 07:05:13.187507 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:13.187517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:13.187565 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:13.187679 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:13.187679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:13.187833 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:13.187879 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.187881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.188512 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.188513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.189032 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.189033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.189367 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.189368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.189408 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.189851 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:13.189856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:13.191655 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":130,"threads_started":1}
W20260927 07:05:13.191835 30731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:13.187423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:34261 (local address 127.21.2.126:40019)
0927 07:05:13.187560 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:13.187562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:13.187573 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:13.187603 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:13.187604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:13.187614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:13.187675 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:13.187945 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.188440 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.189103 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.189300 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.189352 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.191751 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:34261: BlockingRecv error: recv got EOF from 127.21.2.65:34261 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:13.324061 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:13.319300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:50045)
0927 07:05:13.319525 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:13.319537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:13.319630 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:13.319763 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:13.319764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:13.319953 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:13.320002 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.320003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.320715 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.320717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.321242 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.321244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.321633 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.321634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.321701 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.322214 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:13.322219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:13.323964 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":136,"threads_started":1}
W20260927 07:05:13.324110 30731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:13.319409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:50045 (local address 127.21.2.126:40019)
0927 07:05:13.319504 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:13.319506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:13.319610 (+ 104us) server_negotiation.cc:407] Connection header received
0927 07:05:13.319690 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:13.319691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:13.319704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:13.319757 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:13.320083 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.320612 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.321353 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.321549 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.321623 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.324045 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:50045: BlockingRecv error: recv got EOF from 127.21.2.66:50045 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:13 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:13.446581 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:13.441686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:40531)
0927 07:05:13.441857 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:13.441869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:13.441964 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:13.442116 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:13.442117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:13.442292 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:13.442340 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.442342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.443026 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.443027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.443626 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:13.443628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.443995 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.443996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:13.444059 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.444554 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:13.444559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:13.446522 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
W20260927 07:05:13.446677 30731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:13.441763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:40531 (local address 127.21.2.126:40019)
0927 07:05:13.441842 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:13.441844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:13.441946 (+ 102us) server_negotiation.cc:407] Connection header received
0927 07:05:13.442021 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:13.442022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:13.442037 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:13.442110 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:13.442440 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.442940 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.443709 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:13.443909 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:13.443992 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:13.446611 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:40531: BlockingRecv error: recv got EOF from 127.21.2.67:40531 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:14.196573 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:14.192221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:47379)
0927 07:05:14.192372 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:14.192387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:14.192448 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:14.192602 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:14.192602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:14.192759 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:14.192807 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.192808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.193418 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.193418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.193930 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.193931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.194258 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.194259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.194297 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.194738 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:14.194743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:14.196514 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
W20260927 07:05:14.196718 30735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:14.192350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47379 (local address 127.21.2.126:40019)
0927 07:05:14.192483 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:14.192485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:14.192492 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:05:14.192513 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:14.192514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:14.192524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:14.192581 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:14.192867 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.193352 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.194006 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.194195 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.194245 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.196627 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:47379: BlockingRecv error: recv got EOF from 127.21.2.65:47379 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:14.329643 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:14.324608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:59955)
0927 07:05:14.324813 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:14.324827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:14.324930 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:14.325108 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:14.325109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:14.325313 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:14.325364 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.325366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.326192 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.326193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.326740 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.326742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.327144 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.327145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.327206 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.327680 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:14.327685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:14.329541 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260927 07:05:14.329691 30735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:14.324699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:59955 (local address 127.21.2.126:40019)
0927 07:05:14.324798 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:14.324801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:14.324909 (+ 108us) server_negotiation.cc:407] Connection header received
0927 07:05:14.324995 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:14.324996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:14.325011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:14.325089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:14.325452 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.326096 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.326857 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.327054 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.327143 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.329623 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:59955: BlockingRecv error: recv got EOF from 127.21.2.66:59955 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:14 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:14.451834 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:14.447156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:45277)
0927 07:05:14.447332 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:14.447343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:14.447433 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:14.447560 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:14.447561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:14.447722 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:14.447773 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.447775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.448388 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.448389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.448902 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:14.448904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.449209 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.449210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:14.449248 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.449691 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:14.449696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:14.451740 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
W20260927 07:05:14.451884 30735 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:14.447227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:45277 (local address 127.21.2.126:40019)
0927 07:05:14.447325 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:14.447327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:14.447441 (+ 114us) server_negotiation.cc:407] Connection header received
0927 07:05:14.447475 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:14.447476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:14.447488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:14.447538 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:14.447834 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.448316 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.448972 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:14.449145 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:14.449193 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:14.451820 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:45277: BlockingRecv error: recv got EOF from 127.21.2.67:45277 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:15.202652 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:15.197123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:49093)
0927 07:05:15.197303 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:15.197316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:15.197393 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:15.197643 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:15.197645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:15.197827 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:15.197876 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.197877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.198594 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.198595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.199221 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.199241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.199780 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.199781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.199903 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.200446 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:15.200451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:15.202547 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
W20260927 07:05:15.202716 30739 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:15.197209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:49093 (local address 127.21.2.126:40019)
0927 07:05:15.197339 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:15.197341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:15.197376 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:05:15.197501 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:15.197503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:15.197516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:15.197590 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:15.197974 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.198494 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.199341 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.199680 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.199782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.202642 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:49093: BlockingRecv error: recv got EOF from 127.21.2.65:49093 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:15.338503 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:15.330308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:58621)
0927 07:05:15.330463 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:15.330481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:15.330541 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:15.330668 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:15.330669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:15.330837 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:15.330884 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.330886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.331559 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.331560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.332111 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.332113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.332484 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.332485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.332532 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.336424 (+ 3892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:15.336429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:15.338420 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
W20260927 07:05:15.338673 30739 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:15.330404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:58621 (local address 127.21.2.126:40019)
0927 07:05:15.330474 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:15.330475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:15.330541 (+ 66us) server_negotiation.cc:407] Connection header received
0927 07:05:15.330587 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:15.330588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:15.330598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:15.330651 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:15.330947 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.331455 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.332207 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.332402 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.332462 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.338582 (+ 6120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:58621: BlockingRecv error: recv got EOF from 127.21.2.66:58621 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:15 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:15.456952 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:15.452405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:50531)
0927 07:05:15.452540 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:15.452551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:15.452616 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:15.452751 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:15.452751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:15.452917 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:15.452966 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.452967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.453676 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.453677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.454223 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:15.454225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.454575 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.454576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:15.454616 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.455130 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:15.455135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:15.456877 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
W20260927 07:05:15.457098 30739 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:15.452467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:50531 (local address 127.21.2.126:40019)
0927 07:05:15.452541 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:15.452543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:15.452626 (+ 83us) server_negotiation.cc:407] Connection header received
0927 07:05:15.452664 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:15.452664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:15.452678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:15.452729 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:15.453055 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.453578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.454312 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:15.454502 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:15.454563 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:15.457003 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:50531: BlockingRecv error: recv got EOF from 127.21.2.67:50531 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:16.208350 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:16.203323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:44005)
0927 07:05:16.203500 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:16.203513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:16.203574 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:16.203824 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:16.203825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:16.204070 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:16.204135 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.204137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.204870 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.204870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.205496 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.205498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.205824 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.205824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.205864 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.206365 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:16.206372 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:16.208247 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
W20260927 07:05:16.208415 30743 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:16.203468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:44005 (local address 127.21.2.126:40019)
0927 07:05:16.203595 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:16.203598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:16.203605 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:05:16.203637 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:16.203638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:16.203648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:16.203741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:16.204227 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.204803 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.205564 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.205755 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.205812 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.208332 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:44005: BlockingRecv error: recv got EOF from 127.21.2.65:44005 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:16.343465 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:16.338997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:49633)
0927 07:05:16.339161 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:16.339173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:16.339244 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:16.339416 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:16.339417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:16.339587 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:16.339636 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.339638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.340297 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.340298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.340811 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.340812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.341137 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.341138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.341178 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.341667 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:16.341672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:16.343330 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
W20260927 07:05:16.343526 30743 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:16.339083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49633 (local address 127.21.2.126:40019)
0927 07:05:16.339150 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:16.339152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:16.339242 (+ 90us) server_negotiation.cc:407] Connection header received
0927 07:05:16.339289 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:16.339290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:16.339300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:16.339354 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:16.339712 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.340206 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.340880 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.341061 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.341113 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.343480 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:49633: BlockingRecv error: recv got EOF from 127.21.2.66:49633 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:16 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:16.462266 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:16.457463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:55541)
0927 07:05:16.457624 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:16.457639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:16.457713 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:16.457874 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:16.457875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:16.458059 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:16.458107 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.458109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.458798 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.458799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.459346 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:16.459348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.459804 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.459805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:16.459879 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.460401 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:16.460406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:16.462205 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
W20260927 07:05:16.462436 30743 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:16.457593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:55541 (local address 127.21.2.126:40019)
0927 07:05:16.457645 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:16.457647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:16.457692 (+ 45us) server_negotiation.cc:407] Connection header received
0927 07:05:16.457782 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:16.457783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:16.457798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:16.457856 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:16.458190 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.458698 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.459478 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:16.459720 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:16.459792 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:16.462338 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:55541: BlockingRecv error: recv got EOF from 127.21.2.67:55541 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:17.214139 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:17.208883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:55815)
0927 07:05:17.209051 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:17.209063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:17.209126 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:17.209289 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:17.209290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:17.209474 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:17.209520 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.209521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.210238 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.210239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.210778 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.210780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.211159 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.211160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.211220 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.211784 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:17.211790 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:17.214032 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
W20260927 07:05:17.214286 30747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:17.208980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:55815 (local address 127.21.2.126:40019)
0927 07:05:17.209121 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:17.209140 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:17.209151 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:17.209180 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:17.209181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:17.209192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:17.209276 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:17.209600 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.210152 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.210881 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.211085 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.211161 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.214182 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:55815: BlockingRecv error: recv got EOF from 127.21.2.65:55815 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:17.348568 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:17.343994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:49961)
0927 07:05:17.344145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:17.344155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:17.344213 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:17.344387 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:17.344389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:17.344561 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:17.344607 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.344609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.345249 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.345250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.345758 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.345759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.346067 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.346068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.346106 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.346542 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:17.346546 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:17.348494 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260927 07:05:17.348632 30747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:17.344089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49961 (local address 127.21.2.126:40019)
0927 07:05:17.344156 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:17.344158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:17.344219 (+ 61us) server_negotiation.cc:407] Connection header received
0927 07:05:17.344262 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:17.344263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:17.344273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:17.344333 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:17.344685 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.345181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.345827 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.346002 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.346059 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.348587 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:49961: BlockingRecv error: recv got EOF from 127.21.2.66:49961 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:17 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:17.467691 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:17.462840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:43767)
0927 07:05:17.462992 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:17.463005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:17.463080 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:17.463220 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:17.463220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:17.463418 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:17.463481 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.463483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.464122 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.464122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.464664 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:17.464666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.465143 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.465143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:17.465198 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.465709 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:17.465713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:17.467586 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
W20260927 07:05:17.467751 30747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:17.463003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:43767 (local address 127.21.2.126:40019)
0927 07:05:17.463056 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:17.463057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:17.463067 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:05:17.463138 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:17.463139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:17.463151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:17.463191 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:17.463546 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.464027 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.464755 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:17.465048 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:17.465116 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:17.467673 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:43767: BlockingRecv error: recv got EOF from 127.21.2.67:43767 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:18.220428 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:18.214756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:52283)
0927 07:05:18.214908 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:18.214931 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:18.215000 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:18.215336 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:18.215337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:18.215598 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:18.215679 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.215682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.216691 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.216693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.217461 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.217464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.217837 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.217838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.217878 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.218369 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:18.218374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:18.220351 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260927 07:05:18.220619 30751 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:18.214875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:52283 (local address 127.21.2.126:40019)
0927 07:05:18.215030 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:18.215032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:18.215045 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:05:18.215101 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:18.215102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:18.215116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:18.215211 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:18.215773 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.216583 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.217557 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.217757 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.217821 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.220510 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:52283: BlockingRecv error: recv got EOF from 127.21.2.65:52283 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:18.355126 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:18.349146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:41825)
0927 07:05:18.349295 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:18.349306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:18.349398 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:18.349534 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:18.349535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:18.349789 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:18.349837 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.349839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.350522 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.350524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.351068 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.351070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.351475 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.351476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.351538 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.352601 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:18.352605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:18.355037 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260927 07:05:18.355232 30751 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:18.349201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:41825 (local address 127.21.2.126:40019)
0927 07:05:18.349249 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:18.349252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:18.349383 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:05:18.349457 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:18.349458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:18.349471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:18.349520 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:18.349920 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.350426 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.351155 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.351354 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.351474 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.355184 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:41825: BlockingRecv error: recv got EOF from 127.21.2.66:41825 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:18 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:18.473923 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:18.468207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:45651)
0927 07:05:18.468461 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:18.468473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:18.468533 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:18.468672 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:18.468673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:18.468840 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:18.468886 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.468887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.469754 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.469755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.470320 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:18.470322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.470811 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.470812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:18.470914 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.471553 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:18.471559 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:18.473851 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":188,"threads_started":1}
W20260927 07:05:18.474068 30751 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:18.468428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:45651 (local address 127.21.2.126:40019)
0927 07:05:18.468546 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:18.468548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:18.468570 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:05:18.468595 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:18.468596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:18.468610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:18.468666 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:18.468982 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.469582 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.470430 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:18.470709 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:18.470767 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:18.473995 (+ 3228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:45651: BlockingRecv error: recv got EOF from 127.21.2.67:45651 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:19.229208 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:19.221094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:50959)
0927 07:05:19.221228 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:19.221242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:19.221304 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:19.221669 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:19.221670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:19.221978 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:19.222082 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.222085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.223298 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.223299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.224501 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.224505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.225138 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.225140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.225216 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.225850 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:19.225856 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:19.229131 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260927 07:05:19.229480 30755 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:19.221148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:50959 (local address 127.21.2.126:40019)
0927 07:05:19.221269 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:19.221271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:19.221284 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:05:19.221382 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:19.221383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:19.221398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:19.221719 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:19.222201 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.223166 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.224644 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.224977 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.225061 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.229266 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:50959: BlockingRecv error: recv got EOF from 127.21.2.65:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:19.361646 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:19.355908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:47699)
0927 07:05:19.356121 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:19.356135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:19.356213 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:19.356463 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:19.356465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:19.356688 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:19.356759 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.356761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.357653 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.357655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.358241 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.358243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.358759 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.358760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.358817 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.359530 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:19.359536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:19.361573 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260927 07:05:19.361811 30755 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:19.356102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47699 (local address 127.21.2.126:40019)
0927 07:05:19.356159 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:19.356161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:19.356196 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:05:19.356275 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:19.356276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:19.356287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:19.356328 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:19.357010 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.357541 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.358390 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.358631 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.358723 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.361714 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:47699: BlockingRecv error: recv got EOF from 127.21.2.66:47699 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:19 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:19.478991 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:19.474422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:59867)
0927 07:05:19.474599 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:19.474611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:19.474679 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:19.474815 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:19.474816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:19.474986 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:19.475035 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.475038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.475681 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.475682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.476212 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:19.476214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.476619 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.476620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:19.476686 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.477121 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:19.477126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:19.478918 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
W20260927 07:05:19.479139 30755 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:19.474495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:59867 (local address 127.21.2.126:40019)
0927 07:05:19.474576 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:19.474579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:19.474680 (+ 101us) server_negotiation.cc:407] Connection header received
0927 07:05:19.474739 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:19.474740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:19.474752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:19.474795 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:19.475100 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.475615 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.476325 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:19.476520 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:19.476615 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:19.479055 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:59867: BlockingRecv error: recv got EOF from 127.21.2.67:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:20.234390 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:20.229749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:47621)
0927 07:05:20.229923 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:20.229936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:20.229995 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:20.230140 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:20.230140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:20.230305 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:20.230350 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.230351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.231029 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.231030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.231617 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.231619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.232048 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.232049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.232087 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.232571 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:20.232576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:20.234331 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
W20260927 07:05:20.234596 30759 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:20.229854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47621 (local address 127.21.2.126:40019)
0927 07:05:20.229978 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:20.229980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:20.229997 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:05:20.230051 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:20.230052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:20.230062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:20.230120 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:20.230425 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.230948 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.231768 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.231976 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.232038 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.234501 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:47621: BlockingRecv error: recv got EOF from 127.21.2.65:47621 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:20.374837 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:20.362231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:55651)
0927 07:05:20.362399 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:20.362411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:20.362474 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:20.362611 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:20.362612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:20.362776 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:20.362824 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.362826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.363516 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.363517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.364037 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.364039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.364530 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.364532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.364618 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.372727 (+ 8109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:20.372733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:20.374776 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
W20260927 07:05:20.375015 30759 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:20.362389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:55651 (local address 127.21.2.126:40019)
0927 07:05:20.362465 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:20.362467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:20.362476 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:20.362526 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:20.362527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:20.362537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:20.362597 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:20.362890 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.363447 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.364145 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.364350 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.364414 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.374913 (+ 10499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:55651: BlockingRecv error: recv got EOF from 127.21.2.66:55651 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:20 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.67@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:20.484531 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:20.479601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.67:48793)
0927 07:05:20.479749 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:20.479763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:20.479836 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:20.480016 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:20.480017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:20.480206 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:20.480294 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.480297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.481264 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.481264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.481784 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:20.481786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.482130 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.482130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:20.482171 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.482658 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:20.482663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:20.484464 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260927 07:05:20.484602 30759 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:20.479733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48793 (local address 127.21.2.126:40019)
0927 07:05:20.479788 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:20.479790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:20.479816 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:05:20.479923 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:20.479924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:20.479945 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:20.480010 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:20.480423 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.481191 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.481867 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:20.482059 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:20.482118 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:20.484554 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.67:48793: BlockingRecv error: recv got EOF from 127.21.2.67:48793 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 07:05:21 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:21 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.65@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:21.239480 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.234886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.65:38061)
0927 07:05:21.235026 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:21.235040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:21.235094 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:21.235271 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:21.235272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:21.235495 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:21.235548 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.235550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.236193 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.236193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.236713 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.236714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.237068 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.237069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.237107 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.237524 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:21.237528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:21.239324 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
W20260927 07:05:21.239509 30763 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:21.235010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:38061 (local address 127.21.2.126:40019)
0927 07:05:21.235144 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:21.235151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:21.235159 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:05:21.235179 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:21.235180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:21.235190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:21.235256 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:21.235629 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.236126 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.236799 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.236997 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.237057 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.239449 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.65:38061: BlockingRecv error: recv got EOF from 127.21.2.65:38061 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
Sep 27 07:05:21 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:05:21 dist-test-slave-ltqz krb5kdc[29704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.2.66@KRBTEST.COM for oryx/127.21.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:05:21.379585 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.375447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40019 (local address 127.21.2.66:57435)
0927 07:05:21.375558 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:21.375570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:21.375650 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:21.375772 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:21.375773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:21.375922 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:21.375970 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.375971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.376606 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.376606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.377124 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.377126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.377430 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.377431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.377470 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.377897 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:21.377902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:21.379527 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.2.126:40019: Server oryx/127.21.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
W20260927 07:05:21.379684 30763 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:21.375506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:57435 (local address 127.21.2.126:40019)
0927 07:05:21.375562 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:21.375564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:21.375636 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:05:21.375696 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:21.375697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:21.375707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:21.375766 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:21.376024 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.376538 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.377191 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.377370 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.377419 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.379620 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.2.66:57435: BlockingRecv error: recv got EOF from 127.21.2.66:57435 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 07:05:21.407096 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30287
I20260927 07:05:21.411978 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30439
I20260927 07:05:21.416491 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30572
I20260927 07:05:21.421747 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30217
2026-09-27T07:05:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17441 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T07:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:21Z Disabled control of system clock
I20260927 07:05:21.449420 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:33175
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 07:05:21.531229 30768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:21.531451 30768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:21.531472 30768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:21.532706 30768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:21.532755 30768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:21.532768 30768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:21.532780 30768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:21.532792 30768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:21.534350 30768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:33175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:21.534611 30768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:21.534934 30768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:21.539845 30773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:21.539831 30776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:21.540059 30768 server_base.cc:1034] running on GCE node
W20260927 07:05:21.539845 30774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:21.540361 30768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:21.540637 30768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:21.541832 30768 hybrid_clock.cc:648] HybridClock initialized: now 1790492721541816 us; error 34 us; skew 500 ppm
I20260927 07:05:21.543289 30768 webserver.cc:460] Webserver started at http://127.21.2.126:43399/ using document root <none> and password file <none>
I20260927 07:05:21.543588 30768 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:21.543666 30768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:21.543816 30768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:21.544924 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "51d63e683d2f4b05bc708865d9d3db19"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "1f385f3be3cf076cce699e1030d2255d"
server_key_iv: "7c7f992103df0b7f212b609d09ed8bae"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.545307 30768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "51d63e683d2f4b05bc708865d9d3db19"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "1f385f3be3cf076cce699e1030d2255d"
server_key_iv: "7c7f992103df0b7f212b609d09ed8bae"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.546731 30768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:21.547669 30782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.548053 30768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:21.548159 30768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "51d63e683d2f4b05bc708865d9d3db19"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "1f385f3be3cf076cce699e1030d2255d"
server_key_iv: "7c7f992103df0b7f212b609d09ed8bae"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.548249 30768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:21.578740 30768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:21.579108 30768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:21.579273 30768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:21.583184 30768 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:33175
I20260927 07:05:21.583256 30834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:33175 every 8 connection(s)
I20260927 07:05:21.583642 30768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:21.584398 30835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:21.584573 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30768
I20260927 07:05:21.584681 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:05:21.584923 21513 external_mini_cluster.cc:1428] Setting key 35127511c9e52d46e443b43a1af80f77
I20260927 07:05:21.588047 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19: Bootstrap starting.
I20260927 07:05:21.588768 30835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:21.589069 30835 log.cc:824] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:21.589820 30835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19: No bootstrap required, opened a new log
I20260927 07:05:21.590929 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.585502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.21.2.126:33175)
0927 07:05:21.585711 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:21.585718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:21.585743 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:05:21.586316 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:21.586322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:21.586382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:21.586492 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:21.586824 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.587709 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.588784 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.589307 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.589461 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.590180 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:21.590184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:21.590184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:21.590389 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:21.590431 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:21.590445 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:21.590516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:21.590630 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 07:05:21.590713 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260927 07:05:21.592228 30835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } }
I20260927 07:05:21.592360 30835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:21.592381 30835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d63e683d2f4b05bc708865d9d3db19, State: Initialized, Role: FOLLOWER
I20260927 07:05:21.592522 30835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } }
I20260927 07:05:21.592598 30835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:21.592644 30835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:21.592694 30835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:21.593286 30835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } }
I20260927 07:05:21.593405 30835 leader_election.cc:302] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d63e683d2f4b05bc708865d9d3db19; no voters:
I20260927 07:05:21.593583 30835 leader_election.cc:288] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:05:21.593664 30840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:21.593818 30840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [term 1 LEADER]: Becoming Leader. State: Replica: 51d63e683d2f4b05bc708865d9d3db19, State: Running, Role: LEADER
I20260927 07:05:21.593848 30835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:21.593909 30840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } }
I20260927 07:05:21.594266 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51d63e683d2f4b05bc708865d9d3db19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } } }
I20260927 07:05:21.594367 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:21.594269 30842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51d63e683d2f4b05bc708865d9d3db19. Latest consensus state: current_term: 1 leader_uuid: "51d63e683d2f4b05bc708865d9d3db19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d63e683d2f4b05bc708865d9d3db19" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 33175 } } }
I20260927 07:05:21.594447 30842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:21.594652 30849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:05:21.595180 30849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:21.596658 30849 catalog_manager.cc:1347] Generated new cluster ID: 11dd1027440d4e9e9d95e61f1925b8fa
I20260927 07:05:21.596715 30849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:21.619680 30849 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:05:21.620221 30849 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:21.627622 30849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51d63e683d2f4b05bc708865d9d3db19: Generated new TSK 0
I20260927 07:05:21.627847 30849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:05:21.634183 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--builtin_ntp_servers=127.21.2.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 07:05:21.721488 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:21.721724 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:21.721758 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:21.722898 30859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:21.722966 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:21.723011 30859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:21.724448 30859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:21.724727 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:21.725025 30859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:21.730036 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:21.730038 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:21.730159 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:21.730530 30859 server_base.cc:1034] running on GCE node
I20260927 07:05:21.730753 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:21.730995 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:21.732203 30859 hybrid_clock.cc:648] HybridClock initialized: now 1790492721732187 us; error 32 us; skew 500 ppm
I20260927 07:05:21.733615 30859 webserver.cc:460] Webserver started at http://127.21.2.65:38605/ using document root <none> and password file <none>
I20260927 07:05:21.733893 30859 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:21.733968 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:21.734128 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:21.735236 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "4eadc10a07ad4a6b955001c489c4e39e"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "105e886ac28d17b3b8114a01cb754959"
server_key_iv: "a8ab9bd4e5610887a823a0aba1e408a8"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.735630 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "4eadc10a07ad4a6b955001c489c4e39e"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "105e886ac28d17b3b8114a01cb754959"
server_key_iv: "a8ab9bd4e5610887a823a0aba1e408a8"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.737000 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:21.737833 30873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.738037 30859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:21.738139 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "4eadc10a07ad4a6b955001c489c4e39e"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "105e886ac28d17b3b8114a01cb754959"
server_key_iv: "a8ab9bd4e5610887a823a0aba1e408a8"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.738229 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:21.779635 30859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:21.779965 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:21.780120 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:21.780373 30859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:21.780747 30859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:21.780804 30859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.780840 30859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:21.780884 30859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.786438 30859 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:36003
I20260927 07:05:21.786520 30986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:36003 every 8 connection(s)
I20260927 07:05:21.786845 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:05:21.790771 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30859
I20260927 07:05:21.790877 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:05:21.791105 21513 external_mini_cluster.cc:1428] Setting key 3a74a240e8a73d99923b602be15f6373
I20260927 07:05:21.792217 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--builtin_ntp_servers=127.21.2.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 07:05:21.792310 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.787639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:48935 (local address 127.21.2.126:33175)
0927 07:05:21.787690 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:21.787692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:21.787905 (+ 213us) server_negotiation.cc:407] Connection header received
0927 07:05:21.788453 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:21.788454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:21.788470 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:21.788531 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:21.789213 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.789718 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.790739 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.790966 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.791033 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.792020 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:21.792021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:21.792021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:21.792027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:21.792043 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:21.792052 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:21.792113 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:21.792199 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:05:21.792228 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:05:21.792543 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.787598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:33175 (local address 127.21.2.65:48935)
0927 07:05:21.787779 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:21.787813 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:21.788343 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:21.788598 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:21.788602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:21.788612 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:21.789113 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.789117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.789828 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.789829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.790645 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.790648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.791040 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.791041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.791213 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.791868 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:21.791868 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:21.791905 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:21.791937 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:21.792116 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:21.792122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:21.792126 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:05:21.792204 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:05:21.792308 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260927 07:05:21.793161 30987 heartbeater.cc:347] Connected to a master server at 127.21.2.126:33175
I20260927 07:05:21.793258 30987 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:21.793449 30987 heartbeater.cc:511] Master 127.21.2.126:33175 requested a full tablet report, sending...
I20260927 07:05:21.793987 30799 ts_manager.cc:194] Registered new tserver with Master: 4eadc10a07ad4a6b955001c489c4e39e (127.21.2.65:36003)
I20260927 07:05:21.794713 30799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.65:48935
W20260927 07:05:21.873220 30990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:21.873410 30990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:21.873431 30990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:21.874567 30990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:21.874606 30990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:21.874658 30990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:21.876039 30990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:21.876271 30990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:21.876551 30990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:21.881552 30995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:21.881520 30998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:21.881577 30990 server_base.cc:1034] running on GCE node
W20260927 07:05:21.881541 30996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:21.881996 30990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:21.882246 30990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:21.883484 30990 hybrid_clock.cc:648] HybridClock initialized: now 1790492721883471 us; error 34 us; skew 500 ppm
I20260927 07:05:21.884750 30990 webserver.cc:460] Webserver started at http://127.21.2.66:39275/ using document root <none> and password file <none>
I20260927 07:05:21.884944 30990 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:21.884985 30990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:21.885076 30990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:21.886332 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "c9ec57cd4de9449993455a7c3fb7691a"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "710440a741f5c4b3155af35d05caef4d"
server_key_iv: "33f4752895ab332c37adb336cf245012"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.886677 30990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "c9ec57cd4de9449993455a7c3fb7691a"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "710440a741f5c4b3155af35d05caef4d"
server_key_iv: "33f4752895ab332c37adb336cf245012"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.888123 30990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:21.888803 31004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.888978 30990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:21.889088 30990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "c9ec57cd4de9449993455a7c3fb7691a"
format_stamp: "Formatted at 2026-09-27 07:05:21 on dist-test-slave-ltqz"
server_key: "710440a741f5c4b3155af35d05caef4d"
server_key_iv: "33f4752895ab332c37adb336cf245012"
server_key_version: "encryptionkey@0"
I20260927 07:05:21.889168 30990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:21.909989 30990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:21.910338 30990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:21.910503 30990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:21.910755 30990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:21.911096 30990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:21.911154 30990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.911188 30990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:21.911216 30990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:21.916983 30990 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46697
I20260927 07:05:21.917044 31117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46697 every 8 connection(s)
I20260927 07:05:21.917361 30990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:05:21.922183 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.918107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:36763 (local address 127.21.2.126:33175)
0927 07:05:21.918156 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:21.918159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:21.918437 (+ 278us) server_negotiation.cc:407] Connection header received
0927 07:05:21.918887 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:21.918888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:21.918898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:21.918950 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:21.919682 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.920150 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.921048 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.921219 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.921267 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.921997 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:21.921997 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:21.921998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:21.922001 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:21.922011 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:21.922014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:21.922050 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:21.922107 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 07:05:21.922124 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:05:21.922432 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:21.918104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:33175 (local address 127.21.2.66:36763)
0927 07:05:21.918302 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:21.918336 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:21.918808 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:21.919010 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:21.919013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:21.919020 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:21.919610 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.919616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.920228 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.920229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.920977 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:21.920979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:21.921292 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:21.921292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:21.921395 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:21.921885 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:21.921885 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:21.921910 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:21.921931 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:21.922050 (+ 119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:21.922052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:21.922054 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:05:21.922097 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 07:05:21.922177 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 07:05:21.922940 31118 heartbeater.cc:347] Connected to a master server at 127.21.2.126:33175
I20260927 07:05:21.923043 31118 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:21.923223 31118 heartbeater.cc:511] Master 127.21.2.126:33175 requested a full tablet report, sending...
I20260927 07:05:21.923750 30799 ts_manager.cc:194] Registered new tserver with Master: c9ec57cd4de9449993455a7c3fb7691a (127.21.2.66:46697)
I20260927 07:05:21.924127 30799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.66:36763
I20260927 07:05:21.927608 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 30990
I20260927 07:05:21.927685 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:05:21.927832 21513 external_mini_cluster.cc:1428] Setting key 5b2e6a8d6bdfee993f70d9772fe0c567
I20260927 07:05:21.928828 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--builtin_ntp_servers=127.21.2.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 07:05:22.009709 31121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:22.009908 31121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:22.009930 31121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:22.011354 31121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:22.011442 31121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:22.011471 31121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:22.012971 31121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:22.013229 31121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:22.013526 31121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:22.018612 31127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.018740 31121 server_base.cc:1034] running on GCE node
W20260927 07:05:22.018831 31126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.018601 31129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.019079 31121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:22.019328 31121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:22.020514 31121 hybrid_clock.cc:648] HybridClock initialized: now 1790492722020490 us; error 40 us; skew 500 ppm
I20260927 07:05:22.021765 31121 webserver.cc:460] Webserver started at http://127.21.2.67:36327/ using document root <none> and password file <none>
I20260927 07:05:22.022009 31121 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:22.022054 31121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:22.022204 31121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:22.023313 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "3e8c6c8accb64a8f897b3a91486611cf"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "c7025669a56db32b2851056f771aeaf4"
server_key_iv: "639b21cab4cd705cacc10d3c9bb005b6"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.023739 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "3e8c6c8accb64a8f897b3a91486611cf"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "c7025669a56db32b2851056f771aeaf4"
server_key_iv: "639b21cab4cd705cacc10d3c9bb005b6"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.025074 31121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:22.025836 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.026015 31121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:22.026120 31121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "3e8c6c8accb64a8f897b3a91486611cf"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "c7025669a56db32b2851056f771aeaf4"
server_key_iv: "639b21cab4cd705cacc10d3c9bb005b6"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.026198 31121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:22.033272 31121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:22.033528 31121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:22.033679 31121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:22.033887 31121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:22.034193 31121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:22.034246 31121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.034286 31121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:22.034312 31121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.039914 31121 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:33247
I20260927 07:05:22.039986 31248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:33247 every 8 connection(s)
I20260927 07:05:22.040292 31121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:05:22.043493 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31121
I20260927 07:05:22.043565 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:05:22.043780 21513 external_mini_cluster.cc:1428] Setting key ed287c438f479901027b2f455d30c0de
I20260927 07:05:22.045840 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.041008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:51067 (local address 127.21.2.126:33175)
0927 07:05:22.041054 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.041056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.041316 (+ 260us) server_negotiation.cc:407] Connection header received
0927 07:05:22.041827 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.041828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.041839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.041888 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.042568 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:22.043046 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:22.044380 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:22.044597 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:22.044689 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:22.045576 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:22.045578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:22.045578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:22.045583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:22.045598 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:22.045602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:22.045655 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:22.045736 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 07:05:22.045761 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:05:22.046066 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.040990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:33175 (local address 127.21.2.67:51067)
0927 07:05:22.041184 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:22.041228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:22.041697 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:22.041989 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:22.041992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:22.042000 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:22.042484 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:22.042488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:22.043149 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:22.043149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:22.044310 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:22.044313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:22.044700 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:22.044701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:22.044840 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:22.045413 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:22.045413 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:22.045441 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:22.045462 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:22.045658 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:22.045664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:22.045667 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:22.045737 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:05:22.045837 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260927 07:05:22.046532 31249 heartbeater.cc:347] Connected to a master server at 127.21.2.126:33175
I20260927 07:05:22.046619 31249 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:22.046824 31249 heartbeater.cc:511] Master 127.21.2.126:33175 requested a full tablet report, sending...
I20260927 07:05:22.047209 30799 ts_manager.cc:194] Registered new tserver with Master: 3e8c6c8accb64a8f897b3a91486611cf (127.21.2.67:33247)
I20260927 07:05:22.047626 30799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.67:51067
I20260927 07:05:22.055848 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:05:22.057745 30838 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:22.057256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.21.2.126:33175)
0927 07:05:22.057335 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.057337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.057418 (+ 81us) server_negotiation.cc:407] Connection header received
0927 07:05:22.057455 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.057456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.057465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.057520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.057673 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 07:05:22.057785 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30859
I20260927 07:05:22.062562 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30990
I20260927 07:05:22.066768 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31121
I20260927 07:05:22.070931 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 30768
2026-09-27T07:05:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (648 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T07:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:22Z Disabled control of system clock
I20260927 07:05:22.097745 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42155
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:05:22.182186 31260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:22.182410 31260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:22.182432 31260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:22.183729 31260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:22.183774 31260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:22.183787 31260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:22.183801 31260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:22.183849 31260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:22.185221 31260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:42155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:22.185496 31260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:22.185811 31260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:22.190502 31265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.190496 31268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.190508 31266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.190759 31260 server_base.cc:1034] running on GCE node
I20260927 07:05:22.190987 31260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:22.191282 31260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:22.192474 31260 hybrid_clock.cc:648] HybridClock initialized: now 1790492722192473 us; error 41 us; skew 500 ppm
I20260927 07:05:22.193786 31260 webserver.cc:460] Webserver started at http://127.21.2.126:43259/ using document root <none> and password file <none>
I20260927 07:05:22.194067 31260 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:22.194119 31260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:22.194284 31260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:22.195503 31260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "6adcbec69481490a898fc812373ba1f6"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "1a4a7dcf77d9b637a481e1052c661a47"
server_key_iv: "a224cead2e7847e821b701db9a3154f8"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.195887 31260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "6adcbec69481490a898fc812373ba1f6"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "1a4a7dcf77d9b637a481e1052c661a47"
server_key_iv: "a224cead2e7847e821b701db9a3154f8"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.197291 31260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:05:22.198035 31274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.198213 31260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:22.198285 31260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "6adcbec69481490a898fc812373ba1f6"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "1a4a7dcf77d9b637a481e1052c661a47"
server_key_iv: "a224cead2e7847e821b701db9a3154f8"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.198377 31260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:22.201766 31260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:22.202023 31260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:22.202162 31260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:22.206029 31260 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:42155
I20260927 07:05:22.206090 31326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:42155 every 8 connection(s)
I20260927 07:05:22.206372 31260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:22.206990 31327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:22.209812 31327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6: Bootstrap starting.
I20260927 07:05:22.210388 31327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:22.210695 31327 log.cc:824] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:22.211342 31327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6: No bootstrap required, opened a new log
I20260927 07:05:22.211783 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31260
I20260927 07:05:22.211858 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:05:22.212033 21513 external_mini_cluster.cc:1428] Setting key 306057e55df39c1d8eabcb2f064c306d
I20260927 07:05:22.213644 31327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } }
I20260927 07:05:22.213816 31327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:22.213853 31327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adcbec69481490a898fc812373ba1f6, State: Initialized, Role: FOLLOWER
I20260927 07:05:22.214308 31327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } }
I20260927 07:05:22.214426 31327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:22.214463 31327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:22.214515 31327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:22.214890 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.212570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.21.2.126:42155)
0927 07:05:22.212796 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.212805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.212838 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:05:22.213598 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.213615 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.213665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.213817 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.214438 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:22.214443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:22.214443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:22.214448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:22.214499 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:22.214514 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:22.214589 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:22.214661 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 07:05:22.214718 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260927 07:05:22.215534 31327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } }
I20260927 07:05:22.215667 31327 leader_election.cc:302] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6adcbec69481490a898fc812373ba1f6; no voters:
I20260927 07:05:22.215833 31327 leader_election.cc:288] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:05:22.215865 31332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:22.216008 31332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6adcbec69481490a898fc812373ba1f6, State: Running, Role: LEADER
I20260927 07:05:22.216075 31327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:22.216120 31332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } }
I20260927 07:05:22.216462 31334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adcbec69481490a898fc812373ba1f6. Latest consensus state: current_term: 1 leader_uuid: "6adcbec69481490a898fc812373ba1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } } }
I20260927 07:05:22.216459 31333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6adcbec69481490a898fc812373ba1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adcbec69481490a898fc812373ba1f6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42155 } } }
I20260927 07:05:22.216569 31334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:22.216622 31333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:22.216828 31338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:05:22.217345 31338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:22.220412 31338 catalog_manager.cc:1347] Generated new cluster ID: e0a46efced614a42bec11d04fa55d418
I20260927 07:05:22.220469 31338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:22.227849 31338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:05:22.228012 31338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:22.232829 31338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6adcbec69481490a898fc812373ba1f6: Generated new TSK 0
I20260927 07:05:22.233019 31338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:05:22.240347 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--builtin_ntp_servers=127.21.2.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:05:22.321200 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:22.321403 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:22.321427 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:22.322585 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:22.322628 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:22.322654 31351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:22.324039 31351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:22.324285 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:22.324594 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:22.329394 31357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.329393 31356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.329625 31351 server_base.cc:1034] running on GCE node
W20260927 07:05:22.329417 31359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.329913 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:22.330207 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:22.331410 31351 hybrid_clock.cc:648] HybridClock initialized: now 1790492722331389 us; error 32 us; skew 500 ppm
I20260927 07:05:22.332893 31351 webserver.cc:460] Webserver started at http://127.21.2.65:37599/ using document root <none> and password file <none>
I20260927 07:05:22.333217 31351 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:22.333283 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:22.333419 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:22.334766 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "76c2a48359cb4fc0ad02d3d29b96cc88"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "bbbc7adff874ecfc412287a13fdc6404"
server_key_iv: "6e0f8b2e101b06656fa5daa8b56f3552"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.335157 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "76c2a48359cb4fc0ad02d3d29b96cc88"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "bbbc7adff874ecfc412287a13fdc6404"
server_key_iv: "6e0f8b2e101b06656fa5daa8b56f3552"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.336715 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 07:05:22.337512 31365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.337708 31351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:22.337795 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "76c2a48359cb4fc0ad02d3d29b96cc88"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "bbbc7adff874ecfc412287a13fdc6404"
server_key_iv: "6e0f8b2e101b06656fa5daa8b56f3552"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.337882 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:22.341164 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:22.341423 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:22.341566 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:22.341768 31351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:22.342110 31351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:22.342170 31351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.342207 31351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:22.342233 31351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.348170 31351 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:43151
I20260927 07:05:22.348241 31478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:43151 every 8 connection(s)
I20260927 07:05:22.348567 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:05:22.351423 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.349337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:36613 (local address 127.21.2.126:42155)
0927 07:05:22.349410 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.349412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.349619 (+ 207us) server_negotiation.cc:407] Connection header received
0927 07:05:22.350056 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.350057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.350062 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.350122 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.351173 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:22.351175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:22.351175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:22.351179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:22.351194 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:22.351198 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:22.351243 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:22.351307 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 07:05:22.351319 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:05:22.351627 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.349319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42155 (local address 127.21.2.65:36613)
0927 07:05:22.349488 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:22.349527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:22.349977 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:22.350176 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:22.350179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:22.350187 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:22.351033 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:22.351034 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:22.351060 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:22.351086 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:22.351243 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:22.351246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:22.351248 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:05:22.351299 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:05:22.351344 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 07:05:22.352113 31479 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42155
I20260927 07:05:22.352205 31479 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:22.352319 31479 heartbeater.cc:511] Master 127.21.2.126:42155 requested a full tablet report, sending...
I20260927 07:05:22.352800 31291 ts_manager.cc:194] Registered new tserver with Master: 76c2a48359cb4fc0ad02d3d29b96cc88 (127.21.2.65:43151)
I20260927 07:05:22.355137 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31351
I20260927 07:05:22.355216 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:05:22.355461 21513 external_mini_cluster.cc:1428] Setting key 919650f5d25ec6d66b08ad8b15f64e2e
I20260927 07:05:22.356376 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--builtin_ntp_servers=127.21.2.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:05:22.435148 31482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:22.435341 31482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:22.435415 31482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:22.436532 31482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:22.436570 31482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:22.436595 31482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:22.437832 31482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:22.438016 31482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:22.438241 31482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:22.443104 31490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.443245 31487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.443272 31482 server_base.cc:1034] running on GCE node
W20260927 07:05:22.443122 31488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.443730 31482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:22.443979 31482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:22.445132 31482 hybrid_clock.cc:648] HybridClock initialized: now 1790492722445137 us; error 46 us; skew 500 ppm
I20260927 07:05:22.446290 31482 webserver.cc:460] Webserver started at http://127.21.2.66:39383/ using document root <none> and password file <none>
I20260927 07:05:22.446533 31482 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:22.446602 31482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:22.446746 31482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:22.447853 31482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "a0bbfeeda9ee4b8c8a057ea085e5088b"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "13b479f759ed6c6a0af69e5abd52e067"
server_key_iv: "cef0e3d2702ea99f0c2e92c41e619909"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.448211 31482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "a0bbfeeda9ee4b8c8a057ea085e5088b"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "13b479f759ed6c6a0af69e5abd52e067"
server_key_iv: "cef0e3d2702ea99f0c2e92c41e619909"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.449507 31482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:22.450191 31496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.450366 31482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:22.450462 31482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "a0bbfeeda9ee4b8c8a057ea085e5088b"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "13b479f759ed6c6a0af69e5abd52e067"
server_key_iv: "cef0e3d2702ea99f0c2e92c41e619909"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.450548 31482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:22.453867 31482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:22.454123 31482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:22.454264 31482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:22.454473 31482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:22.454777 31482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:22.454831 31482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.454871 31482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:22.454897 31482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.460757 31482 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:33763
I20260927 07:05:22.460798 31609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:33763 every 8 connection(s)
I20260927 07:05:22.461099 31482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:05:22.464211 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.461803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:49025 (local address 127.21.2.126:42155)
0927 07:05:22.461888 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.461891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.462091 (+ 200us) server_negotiation.cc:407] Connection header received
0927 07:05:22.462594 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.462595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.462603 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.462655 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.463931 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:22.463933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:22.463933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:22.463938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:22.463952 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:22.463957 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:22.464028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:22.464124 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:05:22.464141 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:05:22.464391 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.461775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42155 (local address 127.21.2.66:49025)
0927 07:05:22.461974 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:22.462007 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:22.462479 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:22.462734 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:22.462738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:22.462746 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:22.463776 (+ 1030us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:22.463776 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:22.463803 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:22.463830 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:22.464027 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:22.464033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:22.464036 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:22.464131 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 07:05:22.464193 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260927 07:05:22.464872 31610 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42155
I20260927 07:05:22.464959 31610 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:22.465070 31610 heartbeater.cc:511] Master 127.21.2.126:42155 requested a full tablet report, sending...
I20260927 07:05:22.465373 31291 ts_manager.cc:194] Registered new tserver with Master: a0bbfeeda9ee4b8c8a057ea085e5088b (127.21.2.66:33763)
I20260927 07:05:22.470940 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31482
I20260927 07:05:22.471016 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:05:22.471159 21513 external_mini_cluster.cc:1428] Setting key 399e53dd73c7464020dcb4709778ca4d
I20260927 07:05:22.472031 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--builtin_ntp_servers=127.21.2.84:35601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:05:22.551661 31613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:22.551872 31613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:22.551894 31613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:22.553159 31613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:22.553200 31613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:22.553225 31613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:22.554677 31613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:22.554870 31613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:22.555148 31613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:22.560215 31618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.560313 31613 server_base.cc:1034] running on GCE node
W20260927 07:05:22.560240 31621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:22.560211 31619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:22.560789 31613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:22.561003 31613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:22.562166 31613 hybrid_clock.cc:648] HybridClock initialized: now 1790492722562148 us; error 34 us; skew 500 ppm
I20260927 07:05:22.563504 31613 webserver.cc:460] Webserver started at http://127.21.2.67:37205/ using document root <none> and password file <none>
I20260927 07:05:22.563758 31613 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:22.563822 31613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:22.563971 31613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:22.565078 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "8413ff46530243c2ad2c48ac98f8ad29"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "6e8138de3a12e21c2a1e98c8d0d84b07"
server_key_iv: "bb1202287846343fa377cd96542b0ea9"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.565459 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "8413ff46530243c2ad2c48ac98f8ad29"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "6e8138de3a12e21c2a1e98c8d0d84b07"
server_key_iv: "bb1202287846343fa377cd96542b0ea9"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.566852 31613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:22.567633 31627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.567845 31613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:22.567937 31613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "8413ff46530243c2ad2c48ac98f8ad29"
format_stamp: "Formatted at 2026-09-27 07:05:22 on dist-test-slave-ltqz"
server_key: "6e8138de3a12e21c2a1e98c8d0d84b07"
server_key_iv: "bb1202287846343fa377cd96542b0ea9"
server_key_version: "encryptionkey@0"
I20260927 07:05:22.568012 31613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:22.571327 31613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:22.571627 31613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:22.571760 31613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:22.571986 31613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:22.572314 31613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:22.572374 31613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.572417 31613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:22.572443 31613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:22.578344 31613 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:42661
I20260927 07:05:22.578380 31740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:42661 every 8 connection(s)
I20260927 07:05:22.578720 31613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:05:22.581858 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.579462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:42571 (local address 127.21.2.126:42155)
0927 07:05:22.579539 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.579543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.579848 (+ 305us) server_negotiation.cc:407] Connection header received
0927 07:05:22.580275 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.580276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.580281 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.580332 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.581561 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:22.581563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:22.581563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:05:22.581568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:22.581586 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:05:22.581591 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:22.581651 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:22.581786 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 07:05:22.581801 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:05:22.582064 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:22.579463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42155 (local address 127.21.2.67:42571)
0927 07:05:22.579674 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:22.579741 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:22.580193 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:22.580385 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:22.580388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:22.580396 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:22.581369 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:05:22.581370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:22.581398 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:05:22.581427 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:22.581690 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:22.581696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:22.581699 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:22.581783 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 07:05:22.581840 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260927 07:05:22.582567 31741 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42155
I20260927 07:05:22.582656 31741 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:22.582767 31741 heartbeater.cc:511] Master 127.21.2.126:42155 requested a full tablet report, sending...
I20260927 07:05:22.583123 31291 ts_manager.cc:194] Registered new tserver with Master: 8413ff46530243c2ad2c48ac98f8ad29 (127.21.2.67:42661)
I20260927 07:05:22.586615 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31613
I20260927 07:05:22.586722 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:05:22.586912 21513 external_mini_cluster.cc:1428] Setting key 44ab12f41038c8360034b2e2faf2612d
I20260927 07:05:22.587888 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:05:22.589679 31331 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:05:22.589211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.21.2.126:42155)
0927 07:05:22.589273 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:22.589277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:22.589372 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:05:22.589427 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:22.589428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:22.589432 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:22.589482 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:22.589597 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35636: BlockingRecv error: recv got EOF from 127.0.0.1:35636 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:05:22.590015 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31351
I20260927 07:05:22.594975 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31482
I20260927 07:05:22.599113 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31613
I20260927 07:05:22.603206 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31260
2026-09-27T07:05:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (531 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:05:22 dist-test-slave-ltqz krb5kdc[31751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:05:22 dist-test-slave-ltqz krb5kdc[31751](info): set up 2 sockets
Sep 27 07:05:22 dist-test-slave-ltqz krb5kdc[31751](info): commencing operation
krb5kdc: starting...
W20260927 07:05:23.639708 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:05:23 dist-test-slave-ltqz krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:05:23.762257 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40495
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:23.851197 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:23.851575 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:23.851627 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:23.852823 31764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:05:23.852871 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:23.852888 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:23.852903 31764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:05:23.852917 31764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:05:23.854403 31764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:23.854612 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:23.854820 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:23.860118 31770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:23.860132 31772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:23.860119 31769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:23.860375 31764 server_base.cc:1034] running on GCE node
I20260927 07:05:23.860608 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:23.860847 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:23.862002 31764 hybrid_clock.cc:648] HybridClock initialized: now 1790492723861963 us; error 54 us; skew 500 ppm
Sep 27 07:05:23 dist-test-slave-ltqz krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492723, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:23.864557 31764 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:05:23.864969 31764 webserver.cc:460] Webserver started at http://127.21.2.126:43247/ using document root <none> and password file <none>
I20260927 07:05:23.865154 31764 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:23.865195 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:23.865285 31764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:23.866273 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4"
format_stamp: "Formatted at 2026-09-27 07:05:23 on dist-test-slave-ltqz"
server_key: "84a0dc91dba3d6fa34528a1aa5599bd8"
server_key_iv: "f1ff3e8358be20e3ea22775d87aec7a2"
server_key_version: "encryptionkey@0"
I20260927 07:05:23.866578 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4"
format_stamp: "Formatted at 2026-09-27 07:05:23 on dist-test-slave-ltqz"
server_key: "84a0dc91dba3d6fa34528a1aa5599bd8"
server_key_iv: "f1ff3e8358be20e3ea22775d87aec7a2"
server_key_version: "encryptionkey@0"
I20260927 07:05:23.868054 31764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:23.868861 31779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:23.869061 31764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:05:23.869179 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4"
format_stamp: "Formatted at 2026-09-27 07:05:23 on dist-test-slave-ltqz"
server_key: "84a0dc91dba3d6fa34528a1aa5599bd8"
server_key_iv: "f1ff3e8358be20e3ea22775d87aec7a2"
server_key_version: "encryptionkey@0"
I20260927 07:05:23.869262 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:23.880329 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:23.882216 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:23.882375 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:23.886175 31764 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40495
I20260927 07:05:23.886240 31831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40495 every 8 connection(s)
I20260927 07:05:23.886590 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:05:23.887153 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31764
I20260927 07:05:23.887275 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:05:23.887490 21513 external_mini_cluster.cc:1428] Setting key ae8af6bbf189fcd01e78a0308f73b1f2
I20260927 07:05:23.887766 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:23.891223 31832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: Bootstrap starting.
I20260927 07:05:23.891922 31832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:23.892262 31832 log.cc:824] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:23.893069 31832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: No bootstrap required, opened a new log
Sep 27 07:05:23 dist-test-slave-ltqz krb5kdc[31751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:23.894891 31832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } }
I20260927 07:05:23.895068 31832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:23.895108 31832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c9fffc9a9a40c38bc84fd6c64d8fb4, State: Initialized, Role: FOLLOWER
I20260927 07:05:23.895246 31832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } }
I20260927 07:05:23.895339 31832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:05:23.895412 31832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:05:23.895471 31832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:23.896144 31832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } }
I20260927 07:05:23.896268 31832 leader_election.cc:302] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c9fffc9a9a40c38bc84fd6c64d8fb4; no voters:
I20260927 07:05:23.896451 31832 leader_election.cc:288] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:05:23.896494 31837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:23.896624 31837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 12c9fffc9a9a40c38bc84fd6c64d8fb4, State: Running, Role: LEADER
I20260927 07:05:23.896731 31832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:05:23.896704 31837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } }
I20260927 07:05:23.897084 31837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c9fffc9a9a40c38bc84fd6c64d8fb4. Latest consensus state: current_term: 1 leader_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } } }
I20260927 07:05:23.897163 31837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:23.897403 31841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:05:23.897899 31841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:05:23.897970 31839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c9fffc9a9a40c38bc84fd6c64d8fb4" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40495 } } }
I20260927 07:05:23.898049 31839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:05:23.898869 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:23.888029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.21.2.126:40495)
0927 07:05:23.888211 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:23.888220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:23.888251 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:05:23.888815 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:23.888824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:23.888886 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:23.889038 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:23.889586 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:23.890946 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:23.891720 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:23.892362 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:23.892478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:23.894744 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:23.894748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:23.894748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:23.894759 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:23.896641 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:23.897027 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:23.897030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:23.897030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:23.897124 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:23.897422 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:23.897423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:23.897423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:23.897757 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:23.897918 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:23.898551 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 07:05:23.898703 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260927 07:05:23.899708 31841 catalog_manager.cc:1347] Generated new cluster ID: e152e471e55b491d9a762c0a51d29cca
I20260927 07:05:23.899765 31841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:05:23.911962 31841 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:05:23.912712 31841 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:05:23.918556 31841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: Generated new TSK 0
I20260927 07:05:23.918754 31841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:05:23.944455 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:24.025177 31858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:24.025373 31858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:24.025396 31858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:24.026522 31858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:24.026564 31858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:24.026592 31858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:05:24.028131 31858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:24.028388 31858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:24.028607 31858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:24.033457 31866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.033520 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.033499 31864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:24.033661 31858 server_base.cc:1034] running on GCE node
I20260927 07:05:24.033871 31858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:24.034126 31858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:24.035313 31858 hybrid_clock.cc:648] HybridClock initialized: now 1790492724035301 us; error 29 us; skew 500 ppm
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:24.037734 31858 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:05:24.038213 31858 webserver.cc:460] Webserver started at http://127.21.2.65:37541/ using document root <none> and password file <none>
I20260927 07:05:24.038450 31858 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:24.038517 31858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:24.038656 31858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:24.039775 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "91a55f16b61847b9a6dad6396a7b38a3"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "a6ae74b2dc552bbc7fa432186664e2ae"
server_key_iv: "71da2119887af0b226f425735cb892f7"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.040133 31858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "91a55f16b61847b9a6dad6396a7b38a3"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "a6ae74b2dc552bbc7fa432186664e2ae"
server_key_iv: "71da2119887af0b226f425735cb892f7"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.041393 31858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:05:24.042137 31873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.042323 31858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.042416 31858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "91a55f16b61847b9a6dad6396a7b38a3"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "a6ae74b2dc552bbc7fa432186664e2ae"
server_key_iv: "71da2119887af0b226f425735cb892f7"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.042500 31858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:24.057107 31858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:24.059001 31858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:24.059161 31858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:24.059499 31858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:24.059824 31858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:24.059877 31858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.059921 31858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:24.059947 31858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.066080 31858 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:35661
I20260927 07:05:24.066138 31986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:35661 every 8 connection(s)
I20260927 07:05:24.066526 31858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:05:24.068964 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31858
I20260927 07:05:24.069088 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:05:24.069288 21513 external_mini_cluster.cc:1428] Setting key 8c845e98f67f0196558e18324c4ec884
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:24.076661 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.067252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47293 (local address 127.21.2.126:40495)
0927 07:05:24.067328 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.067331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.067583 (+ 252us) server_negotiation.cc:407] Connection header received
0927 07:05:24.068059 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.068060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.068070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.068121 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:24.069035 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.069708 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.070744 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.071080 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.071156 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.073733 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:24.073735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:24.073735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:24.073742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:24.075222 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.075685 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.075686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.075686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.075748 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.076002 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.076003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.076003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.076120 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:24.076194 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.076468 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.076568 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:05:24.076893 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.067197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40495 (local address 127.21.2.65:47293)
0927 07:05:24.067435 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.067483 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.067976 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.068172 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.068176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.068744 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:24.068934 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.068938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.069801 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.069802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.070647 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.070650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.071184 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.071185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.071358 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.071896 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:24.071906 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:24.073610 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:24.075324 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.075331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.075333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.075594 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.075815 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.075816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.075816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.075929 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.076188 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:24.076195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:24.076335 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.076437 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.076591 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:05:24.077498 31987 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40495
I20260927 07:05:24.077584 31987 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:24.077730 31987 heartbeater.cc:511] Master 127.21.2.126:40495 requested a full tablet report, sending...
I20260927 07:05:24.078320 31796 ts_manager.cc:194] Registered new tserver with Master: 91a55f16b61847b9a6dad6396a7b38a3 (127.21.2.65:35661)
I20260927 07:05:24.078935 31796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:47293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:05:24.095878 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:24.176775 31992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:24.177006 31992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:24.177045 31992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:24.178161 31992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:24.178231 31992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:24.178279 31992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:05:24.179770 31992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:24.180032 31992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:24.180259 31992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:05:24.185241 31992 server_base.cc:1034] running on GCE node
W20260927 07:05:24.185220 31998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.185209 31997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.185200 32000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:24.185590 31992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:24.185837 31992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:24.187026 31992 hybrid_clock.cc:648] HybridClock initialized: now 1790492724187006 us; error 31 us; skew 500 ppm
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:24.189893 31992 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:05:24.190447 31992 webserver.cc:460] Webserver started at http://127.21.2.66:34625/ using document root <none> and password file <none>
I20260927 07:05:24.190738 31992 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:24.190805 31992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:24.190944 31992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:24.192124 31992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "10b22db509054ca8aecd6edf5892b6d7"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "e82c23070a596f8592be79a365712763"
server_key_iv: "a14a1c3dee63f16f3a302310d83aaef3"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.192430 31992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "10b22db509054ca8aecd6edf5892b6d7"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "e82c23070a596f8592be79a365712763"
server_key_iv: "a14a1c3dee63f16f3a302310d83aaef3"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.193670 31992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:05:24.194350 32007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.194576 31992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:05:24.194656 31992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "10b22db509054ca8aecd6edf5892b6d7"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "e82c23070a596f8592be79a365712763"
server_key_iv: "a14a1c3dee63f16f3a302310d83aaef3"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.194746 31992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:24.203059 31992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:24.204838 31992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:24.204986 31992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:24.205193 31992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:24.205484 31992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:24.205524 31992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.205546 31992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:24.205591 31992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.211171 31992 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:35081
I20260927 07:05:24.211236 32120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:35081 every 8 connection(s)
I20260927 07:05:24.211570 31992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:24.220347 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.212279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:37185 (local address 127.21.2.126:40495)
0927 07:05:24.212369 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.212371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.212532 (+ 161us) server_negotiation.cc:407] Connection header received
0927 07:05:24.212985 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.212987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.212999 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.213052 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:24.213946 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.214429 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.215441 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.215688 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.215746 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.217787 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:24.217789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:24.217790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:24.217798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:24.218972 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.219391 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.219392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.219393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.219451 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.219708 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.219709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.219709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.219826 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:24.219902 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.220172 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.220254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 07:05:24.220536 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.212244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40495 (local address 127.21.2.66:37185)
0927 07:05:24.212416 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.212450 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.212906 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.213104 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.213107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.213651 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:24.213849 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.213853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.214507 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.214508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.215289 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.215291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.215772 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.215773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.215873 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.216318 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:24.216326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:24.217671 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:24.219063 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.219070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.219072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.219291 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.219524 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.219525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.219526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.219639 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.219894 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:24.219900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:24.220044 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.220151 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.220305 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260927 07:05:24.221045 32121 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40495
I20260927 07:05:24.221086 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 31992
I20260927 07:05:24.221124 32121 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:24.221210 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:05:24.221344 32121 heartbeater.cc:511] Master 127.21.2.126:40495 requested a full tablet report, sending...
I20260927 07:05:24.221444 21513 external_mini_cluster.cc:1428] Setting key c206092d207345afb89453894f5b0d49
I20260927 07:05:24.221768 31796 ts_manager.cc:194] Registered new tserver with Master: 10b22db509054ca8aecd6edf5892b6d7 (127.21.2.66:35081)
I20260927 07:05:24.222289 31796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:37185
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:05:24.246347 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:33835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:05:24.326138 32126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:05:24.326382 32126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:05:24.326421 32126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:05:24.327595 32126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:05:24.327664 32126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:05:24.327709 32126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:05:24.329145 32126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:33835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:05:24.329413 32126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:05:24.329645 32126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:05:24.334524 32131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.334581 32132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:05:24.334595 32134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:05:24.334782 32126 server_base.cc:1034] running on GCE node
I20260927 07:05:24.335005 32126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:05:24.335238 32126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:05:24.336402 32126 hybrid_clock.cc:648] HybridClock initialized: now 1790492724336381 us; error 38 us; skew 500 ppm
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:05:24.338975 32126 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:05:24.339566 32126 webserver.cc:460] Webserver started at http://127.21.2.67:33637/ using document root <none> and password file <none>
I20260927 07:05:24.339758 32126 fs_manager.cc:359] Metadata directory not provided
I20260927 07:05:24.339798 32126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:05:24.339885 32126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:05:24.340919 32126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "274e3f5aad4d4f26b6738537850384a4"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "08a64dbba11d873e9b7f0cf6b0c8144d"
server_key_iv: "c081300aa605a888f7c6e20f8bec0d12"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.341226 32126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "274e3f5aad4d4f26b6738537850384a4"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "08a64dbba11d873e9b7f0cf6b0c8144d"
server_key_iv: "c081300aa605a888f7c6e20f8bec0d12"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.342447 32126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:05:24.343125 32141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.343291 32126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:05:24.343405 32126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "274e3f5aad4d4f26b6738537850384a4"
format_stamp: "Formatted at 2026-09-27 07:05:24 on dist-test-slave-ltqz"
server_key: "08a64dbba11d873e9b7f0cf6b0c8144d"
server_key_iv: "c081300aa605a888f7c6e20f8bec0d12"
server_key_version: "encryptionkey@0"
I20260927 07:05:24.343492 32126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:05:24.351044 32126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:05:24.353055 32126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:05:24.353219 32126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:05:24.353430 32126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:05:24.353739 32126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:05:24.353798 32126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.353839 32126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:05:24.353866 32126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:05:24.359550 32126 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:40251
I20260927 07:05:24.359635 32254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:40251 every 8 connection(s)
I20260927 07:05:24.359949 32126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:05:24.360944 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 32126
I20260927 07:05:24.361066 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:05:24.361258 21513 external_mini_cluster.cc:1428] Setting key 228c67918b37ad14b15526dc9ae23e67
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[31751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492724, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:05:24.370294 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.360704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:59439 (local address 127.21.2.126:40495)
0927 07:05:24.360791 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.360794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.360949 (+ 155us) server_negotiation.cc:407] Connection header received
0927 07:05:24.361620 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.361621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.361632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.361714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:24.362762 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.363249 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.364274 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.364459 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.364516 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.366571 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:24.366572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:24.366572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:24.366579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:24.369035 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.369422 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.369423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.369424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.369475 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.369745 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.369746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.369747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.369865 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:24.369928 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.370153 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.370221 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 07:05:24.370493 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.360661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40495 (local address 127.21.2.67:59439)
0927 07:05:24.360823 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.360863 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.361500 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.361801 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.361805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.362482 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:05:24.362665 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.362669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.363327 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.363328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.364194 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.364196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.364533 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.364534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.364628 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.365086 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:05:24.365094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:05:24.366382 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:05:24.369122 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.369128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.369129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.369355 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.369543 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:05:24.369543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:05:24.369544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:05:24.369663 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:05:24.369929 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:05:24.369933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:05:24.370044 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.370125 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.370251 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260927 07:05:24.371089 32255 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40495
I20260927 07:05:24.371197 32255 heartbeater.cc:464] Registering TS with master...
I20260927 07:05:24.371429 32255 heartbeater.cc:511] Master 127.21.2.126:40495 requested a full tablet report, sending...
I20260927 07:05:24.371816 31796 ts_manager.cc:194] Registered new tserver with Master: 274e3f5aad4d4f26b6738537850384a4 (127.21.2.67:40251)
I20260927 07:05:24.372198 31796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:59439
I20260927 07:05:24.372890 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:05:24.380139 31835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.374188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.21.2.126:40495)
0927 07:05:24.374236 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.374238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.374299 (+ 61us) server_negotiation.cc:407] Connection header received
0927 07:05:24.374373 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.374374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.374384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.374437 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:05:24.375009 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.375533 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.376296 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.376494 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.376568 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.377781 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:05:24.377782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:05:24.377783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:05:24.377789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:05:24.378925 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.379276 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.379277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.379278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.379338 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:05:24.379582 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:05:24.379583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:05:24.379584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:05:24.379743 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:05:24.379856 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.379961 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.380040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:05:24.381536 31796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:05:24.383056 31796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:05:24.389883 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:35081 (local address 127.0.0.1:34212)
0927 07:05:24.385476 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.385486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.385536 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.385989 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.385989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.385991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:24.386066 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.386067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.387333 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.387333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.388760 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.388763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.389651 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.389652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.389704 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.389707 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.389765 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.389798 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":36,"thread_start_us":70,"threads_started":1}
I20260927 07:05:24.389799 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:40251 (local address 127.0.0.1:55128)
0927 07:05:24.385338 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.385378 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.385447 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.385902 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.385905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.385907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:24.385987 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.385989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.387332 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.387333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.388828 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.388829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.389478 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.389479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.389557 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.389569 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.389640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.389707 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":92,"thread_start_us":57,"threads_started":1}
I20260927 07:05:24.389977 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.21.2.67:40251)
0927 07:05:24.385663 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.385669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.385680 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:24.385720 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.385723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.385740 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.385817 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:24.386073 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.387204 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.388898 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.389388 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.389477 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.389537 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.389841 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.389887 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":81,"threads_started":1}
I20260927 07:05:24.389897 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.21.2.66:35081)
0927 07:05:24.385818 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.385823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.385834 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:05:24.385875 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.385877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.385893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.385975 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:24.386155 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.387235 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.388851 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.389576 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.389642 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.389696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.389778 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.389817 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":94,"threads_started":1}
I20260927 07:05:24.390563 32189 tablet_service.cc:1467] Processing CreateTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 (DEFAULT_TABLE table=test-table [id=f86e98c045a04632bca7ae7a9a6a6c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:24.390645 32055 tablet_service.cc:1467] Processing CreateTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 (DEFAULT_TABLE table=test-table [id=f86e98c045a04632bca7ae7a9a6a6c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:24.390939 32189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f7d3cd10ee40ab9708eb6aa444eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:24.390985 32055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f7d3cd10ee40ab9708eb6aa444eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:24.391637 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:35661 (local address 127.0.0.1:52958)
0927 07:05:24.385628 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.385636 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.385679 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.387351 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.387352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.387354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:24.387423 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.387425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.388445 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.388446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.389415 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.389418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.391396 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.391397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.391461 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.391466 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.391519 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.391562 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":64,"threads_started":1}
I20260927 07:05:24.392007 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.385647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.21.2.65:35661)
0927 07:05:24.385831 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.385835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.385844 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:24.385878 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.385880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.385893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.385976 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:24.387503 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.388348 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.389517 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.391203 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.391269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.391321 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.391892 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.391933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260927 07:05:24.392794 31921 tablet_service.cc:1467] Processing CreateTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 (DEFAULT_TABLE table=test-table [id=f86e98c045a04632bca7ae7a9a6a6c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:05:24.393093 31921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f7d3cd10ee40ab9708eb6aa444eea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:05:24.393946 32269 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Bootstrap starting.
I20260927 07:05:24.394444 32270 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Bootstrap starting.
I20260927 07:05:24.394706 32269 tablet_bootstrap.cc:654] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:24.395120 32270 tablet_bootstrap.cc:654] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:24.395306 32269 log.cc:824] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:24.395706 32270 log.cc:824] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:24.396138 32273 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: Bootstrap starting.
I20260927 07:05:24.396373 32269 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: No bootstrap required, opened a new log
I20260927 07:05:24.396375 32270 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: No bootstrap required, opened a new log
I20260927 07:05:24.396445 32270 ts_tablet_manager.cc:1397] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:05:24.396445 32269 ts_tablet_manager.cc:1397] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:05:24.396855 32273 tablet_bootstrap.cc:654] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:05:24.397523 32273 log.cc:824] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:05:24.398309 32273 tablet_bootstrap.cc:492] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: No bootstrap required, opened a new log
I20260927 07:05:24.398389 32273 ts_tablet_manager.cc:1397] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:05:24.398572 32270 raft_consensus.cc:348] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.398747 32270 raft_consensus.cc:374] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:24.398790 32270 raft_consensus.cc:729] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b22db509054ca8aecd6edf5892b6d7, State: Initialized, Role: FOLLOWER
I20260927 07:05:24.398805 32269 raft_consensus.cc:348] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.398895 32270 consensus_queue.cc:260] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.398941 32269 raft_consensus.cc:374] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:24.398973 32269 raft_consensus.cc:729] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274e3f5aad4d4f26b6738537850384a4, State: Initialized, Role: FOLLOWER
I20260927 07:05:24.399094 32269 consensus_queue.cc:260] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.399168 32270 ts_tablet_manager.cc:1428] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:05:24.399396 32255 heartbeater.cc:503] Master 127.21.2.126:40495 was elected leader, sending a full tablet report...
I20260927 07:05:24.399446 32121 heartbeater.cc:503] Master 127.21.2.126:40495 was elected leader, sending a full tablet report...
I20260927 07:05:24.399403 32269 ts_tablet_manager.cc:1428] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:05:24.400658 32273 raft_consensus.cc:348] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.400830 32273 raft_consensus.cc:374] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:05:24.400875 32273 raft_consensus.cc:729] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a55f16b61847b9a6dad6396a7b38a3, State: Initialized, Role: FOLLOWER
I20260927 07:05:24.400982 32273 consensus_queue.cc:260] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.401263 32273 ts_tablet_manager.cc:1428] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:05:24.401285 31987 heartbeater.cc:503] Master 127.21.2.126:40495 was elected leader, sending a full tablet report...
I20260927 07:05:24.413321 32277 raft_consensus.cc:482] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:05:24.413451 32277 raft_consensus.cc:504] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.413806 32277 leader_election.cc:288] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274e3f5aad4d4f26b6738537850384a4 (127.21.2.67:40251), 10b22db509054ca8aecd6edf5892b6d7 (127.21.2.66:35081)
I20260927 07:05:24.416688 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.414079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:55353 (local address 127.21.2.66:35081)
0927 07:05:24.414130 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.414131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.414262 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:05:24.414289 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.414290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.414300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.414364 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:24.414520 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.414984 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.415960 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.416336 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.416421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.416431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.416561 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.416604 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:05:24.416706 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.413958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:40251 (local address 127.21.2.65:43693)
0927 07:05:24.414015 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.414029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.414080 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.414283 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.414284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.414287 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:24.414358 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.414360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.415005 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.415006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.415871 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.415874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.416427 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.416428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.416512 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.416515 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.416585 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.416621 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 07:05:24.416720 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.414013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:43693 (local address 127.21.2.67:40251)
0927 07:05:24.414070 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.414072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.414082 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:05:24.414175 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.414177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.414203 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.414260 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:05:24.414435 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.414930 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.415961 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.416335 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.416416 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.416427 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.416610 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.416653 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 07:05:24.416971 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.413958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:35081 (local address 127.21.2.65:55353)
0927 07:05:24.414173 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.414183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:05:24.414237 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:05:24.414371 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:05:24.414371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:05:24.414373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:05:24.414456 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.414458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.415068 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.415069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.415870 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:05:24.415872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.416421 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.416422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:05:24.416493 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.416496 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:05:24.416538 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 07:05:24.416599 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":48,"thread_start_us":115,"threads_started":1}
I20260927 07:05:24.417018 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f7d3cd10ee40ab9708eb6aa444eea1" candidate_uuid: "91a55f16b61847b9a6dad6396a7b38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274e3f5aad4d4f26b6738537850384a4" is_pre_election: true
I20260927 07:05:24.417189 32209 raft_consensus.cc:2454] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a55f16b61847b9a6dad6396a7b38a3 in term 0.
I20260927 07:05:24.417313 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f7d3cd10ee40ab9708eb6aa444eea1" candidate_uuid: "91a55f16b61847b9a6dad6396a7b38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b22db509054ca8aecd6edf5892b6d7" is_pre_election: true
I20260927 07:05:24.417419 31877 leader_election.cc:302] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274e3f5aad4d4f26b6738537850384a4, 91a55f16b61847b9a6dad6396a7b38a3; no voters:
I20260927 07:05:24.417478 32075 raft_consensus.cc:2454] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a55f16b61847b9a6dad6396a7b38a3 in term 0.
I20260927 07:05:24.417532 32277 raft_consensus.cc:2781] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:05:24.417599 32277 raft_consensus.cc:482] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:05:24.417637 32277 raft_consensus.cc:3037] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:24.418361 32277 raft_consensus.cc:504] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.418507 32277 leader_election.cc:288] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [CANDIDATE]: Term 1 election: Requested vote from peers 274e3f5aad4d4f26b6738537850384a4 (127.21.2.67:40251), 10b22db509054ca8aecd6edf5892b6d7 (127.21.2.66:35081)
I20260927 07:05:24.418679 32209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f7d3cd10ee40ab9708eb6aa444eea1" candidate_uuid: "91a55f16b61847b9a6dad6396a7b38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274e3f5aad4d4f26b6738537850384a4"
I20260927 07:05:24.418722 32075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f7d3cd10ee40ab9708eb6aa444eea1" candidate_uuid: "91a55f16b61847b9a6dad6396a7b38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b22db509054ca8aecd6edf5892b6d7"
I20260927 07:05:24.418757 32209 raft_consensus.cc:3037] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:24.418820 32075 raft_consensus.cc:3037] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:05:24.419602 32209 raft_consensus.cc:2454] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a55f16b61847b9a6dad6396a7b38a3 in term 1.
I20260927 07:05:24.419616 32075 raft_consensus.cc:2454] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a55f16b61847b9a6dad6396a7b38a3 in term 1.
I20260927 07:05:24.419757 31877 leader_election.cc:302] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274e3f5aad4d4f26b6738537850384a4, 91a55f16b61847b9a6dad6396a7b38a3; no voters:
I20260927 07:05:24.419905 32277 raft_consensus.cc:2781] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:05:24.420046 32277 raft_consensus.cc:686] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [term 1 LEADER]: Becoming Leader. State: Replica: 91a55f16b61847b9a6dad6396a7b38a3, State: Running, Role: LEADER
I20260927 07:05:24.420145 32277 consensus_queue.cc:237] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } }
I20260927 07:05:24.420893 31795 catalog_manager.cc:5495] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91a55f16b61847b9a6dad6396a7b38a3 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "91a55f16b61847b9a6dad6396a7b38a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91a55f16b61847b9a6dad6396a7b38a3" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 35661 } health_report { overall_health: HEALTHY } } }
I20260927 07:05:24.434489 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.432315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.21.2.65:35661)
0927 07:05:24.432378 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:05:24.432380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:05:24.432389 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:05:24.432429 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:05:24.432430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:05:24.432447 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:05:24.432507 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:05:24.432625 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.433073 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.433791 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:05:24.433996 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:05:24.434079 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:05:24.434198 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:05:24.434298 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:05:24.434366 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:05:24.434385 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 07:05:24.434405 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:05:24.436344 32075 raft_consensus.cc:1261] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 1 FOLLOWER]: Refusing update from remote peer 91a55f16b61847b9a6dad6396a7b38a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:24.436347 32209 raft_consensus.cc:1261] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 1 FOLLOWER]: Refusing update from remote peer 91a55f16b61847b9a6dad6396a7b38a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:05:24.436745 32277 consensus_queue.cc:1035] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274e3f5aad4d4f26b6738537850384a4" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:24.436934 32279 consensus_queue.cc:1035] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 91a55f16b61847b9a6dad6396a7b38a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b22db509054ca8aecd6edf5892b6d7" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:05:24.439211 32282 mvcc.cc:205] Tried to move back new op lower bound from 7333858199288664064 to 7333858199225487360. Current Snapshot: MvccSnapshot[applied={T|T < 7333858199288664064}]
I20260927 07:05:24.439731 32283 mvcc.cc:205] Tried to move back new op lower bound from 7333858199288664064 to 7333858199225487360. Current Snapshot: MvccSnapshot[applied={T|T < 7333858199288664064}]
I20260927 07:05:24.439776 32284 mvcc.cc:205] Tried to move back new op lower bound from 7333858199288664064 to 7333858199225487360. Current Snapshot: MvccSnapshot[applied={T|T < 7333858199288664064}]
I20260927 07:05:24.441623 31796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:24.441732 31796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:05:24.442488 31796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12c9fffc9a9a40c38bc84fd6c64d8fb4: Sending DeleteTablet for 3 replicas of tablet 68f7d3cd10ee40ab9708eb6aa444eea1
I20260927 07:05:24.442852 32055 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34212
I20260927 07:05:24.442867 31921 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:52958
I20260927 07:05:24.443076 32189 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f7d3cd10ee40ab9708eb6aa444eea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:05:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55128
I20260927 07:05:24.443286 32294 tablet_replica.cc:330] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: stopping tablet replica
I20260927 07:05:24.443418 32294 raft_consensus.cc:2229] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:24.443526 32294 raft_consensus.cc:2258] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:24.443629 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31858
I20260927 07:05:24.444639 32294 ts_tablet_manager.cc:1905] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:24.444795 32292 tablet_replica.cc:330] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: stopping tablet replica
I20260927 07:05:24.444883 32292 raft_consensus.cc:2229] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:05:24.444971 32292 raft_consensus.cc:2258] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:05:24.445525 32292 ts_tablet_manager.cc:1905] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:05:24.446487 32294 ts_tablet_manager.cc:1918] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:24.446578 32294 log.cc:1196] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/68f7d3cd10ee40ab9708eb6aa444eea1
I20260927 07:05:24.446862 32294 ts_tablet_manager.cc:1939] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 274e3f5aad4d4f26b6738537850384a4: Deleting consensus metadata
I20260927 07:05:24.447289 31783 catalog_manager.cc:4841] TS 274e3f5aad4d4f26b6738537850384a4 (127.21.2.67:40251): tablet 68f7d3cd10ee40ab9708eb6aa444eea1 (table test-table [id=f86e98c045a04632bca7ae7a9a6a6c36]) successfully deleted
I20260927 07:05:24.447623 32292 ts_tablet_manager.cc:1918] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:05:24.447767 32292 log.cc:1196] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/68f7d3cd10ee40ab9708eb6aa444eea1
I20260927 07:05:24.448014 32292 ts_tablet_manager.cc:1939] T 68f7d3cd10ee40ab9708eb6aa444eea1 P 10b22db509054ca8aecd6edf5892b6d7: Deleting consensus metadata
I20260927 07:05:24.448379 31780 catalog_manager.cc:4841] TS 10b22db509054ca8aecd6edf5892b6d7 (127.21.2.66:35081): tablet 68f7d3cd10ee40ab9708eb6aa444eea1 (table test-table [id=f86e98c045a04632bca7ae7a9a6a6c36]) successfully deleted
W20260927 07:05:24.450829 31783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:35661 (error 108)
I20260927 07:05:24.451381 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31992
I20260927 07:05:24.451428 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:05:24.451245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:35661 (local address 127.0.0.1:52970)
0927 07:05:24.451305 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:05:24.451348 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 07:05:24.451578 31783 catalog_manager.cc:4584] TS 91a55f16b61847b9a6dad6396a7b38a3 (127.21.2.65:35661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68f7d3cd10ee40ab9708eb6aa444eea1: Network error: Client connection negotiation failed: client connection to 127.21.2.65:35661: connect: Connection refused (error 111)
I20260927 07:05:24.457573 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 32126
I20260927 07:05:24.462344 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 31764
2026-09-27T07:05:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1860 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[32298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[32298](info): set up 2 sockets
Sep 27 07:05:24 dist-test-slave-ltqz krb5kdc[32298](info): commencing operation
krb5kdc: starting...
W20260927 07:05:25.483664 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:05:25 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:05:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangeradmin_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangeradmin_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangerlookup_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangerlookup_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
I20260927 07:05:25.646366 21513 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHv0k7x/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/postgres -l logfile start
2026-09-27 07:05:28.140 UTC [32328] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:05:28.140 UTC [32328] LOG: listening on IPv4 address "127.21.2.84", port 51495
2026-09-27 07:05:28.143 UTC [32328] LOG: listening on Unix socket "/tmp/.s.PGSQL.51495"
2026-09-27 07:05:28.166 UTC [32331] LOG: database system was shut down at 2026-09-27 07:05:26 UTC
2026-09-27 07:05:28.170 UTC [32328] LOG: database system is ready to accept connections
I20260927 07:05:29.128129 21513 mini_postgres.cc:96] Postgres bound to 51495
2026-09-27 07:05:29.134 UTC [32339] FATAL: database "slave" does not exist
127.21.2.84:51495 - accepting connections
I20260927 07:05:29.134981 21513 mini_ranger.cc:161] Starting Ranger...
I20260927 07:05:29.150379 21513 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:05:29.629041 21513 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:05:29.629109 21513 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin
2026-09-27 07:05:29,937 [I] DB FLAVOR :POSTGRES
2026-09-27 07:05:29,937 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:05:29,937 [I] Checking connection..
2026-09-27 07:05:29,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:05:30,197 [I] Checking connection passed.
2026-09-27 07:05:30,197 [I] --------- Verifying version history table ---------
2026-09-27 07:05:30,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:05:30,454 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:05:30,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:05:30,702 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:05:30,702 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:05:30,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:05:30.940 UTC [32403] WARNING: there is no transaction in progress
2026-09-27 07:05:30,946 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:05:30,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:05:31,177 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:05:31,177 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:05:31,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:05:31,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:05:31,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ltqz.c.gcp-upstream.internal','N') ;"
2026-09-27 07:05:31,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:05:32,120 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:05:32,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:05:32,367 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:05:32,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:05:32,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:05:32,872 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:05:32,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:05:33.819 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.836 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.853 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.869 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.886 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.904 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.960 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.967 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.982 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:33.993 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.004 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.016 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.026 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.033 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.040 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34.047 UTC [32511] WARNING: there is no transaction in progress
2026-09-27 07:05:34,712 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:05:34,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:05:34,938 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:05:34,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:05:35,186 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:05:35,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:05:35,448 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:05:35,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:05:35,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:05:35,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ltqz.c.gcp-upstream.internal';"
2026-09-27 07:05:36,170 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:05:36,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:05:36,443 [I] DB_PATCHES have already been applied
I20260927 07:05:36.449527 21513 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHv0k7x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:05:36.449615 21513 mini_ranger.cc:193] Using host: 127.21.2.84
I20260927 07:05:36.453141 21513 mini_ranger.cc:227] Ranger admin URL: http://127.21.2.84:41243
Sep 27, 2026 7:05:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:05:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:05:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:05:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:05:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:05:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:05:37.558881 21513 mini_ranger.cc:160] Time spent starting Ranger: real 8.424s user 0.002s sys 0.007s
Sep 27, 2026 7:05:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:05:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 7:05:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:05:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:05:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:05:59.696084 21513 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangerkms_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/rangerkms_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.2.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.2.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:05:59 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492759, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:05:59.781371 21513 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:05:59.790856 21513 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:06:00.623301 21513 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:06:00.623423 21513 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms
I20260927 07:06:00.623449 21513 mini_ranger_kms.cc:227] Using postgres at 127.21.2.84:51495
2026-09-27 07:06:00,788 [I] DB FLAVOR :POSTGRES
2026-09-27 07:06:00,788 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:06:00,788 [I] Checking connection
2026-09-27 07:06:00,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:06:01,023 [I] connection success
2026-09-27 07:06:01,023 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:06:01,023 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:06:01,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:06:01,285 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:06:01,286 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:06:01,286 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:06:01,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:51495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:06:01,561 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:06:02.027884 21513 mini_ranger_kms.cc:310] Created kms service
I20260927 07:06:02.226948 21513 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:06:02.269512 21513 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:06:02 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492759, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:02.623808 21513 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:06:02.623924 21513 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:06:02.623947 21513 mini_ranger_kms.cc:240] Using host: 127.21.2.84
I20260927 07:06:02.627781 21513 mini_ranger_kms.cc:276] Ranger KMS PID: 32691
I20260927 07:06:02.627904 21513 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.2.84:45811
Sep 27, 2026 7:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:06:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:06:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 07:06:04.935447 21513 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.154s user 0.000s sys 0.147s
I20260927 07:06:04.935563 21513 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:06:04.935647 21513 mini_ranger_kms.cc:401] 127.21.2.84:45811/kms/v1/keys
Sep 27 07:06:05 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492765, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 07:06:05 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492765, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
Sep 27 07:06:05 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492765, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
Sep 27 07:06:07 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:06:07.927923 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:41957
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:41957
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:45811/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:08.023799 32742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:08.024010 32742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:08.024032 32742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:08.025213 32742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:08.025260 32742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:08.025274 32742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:08.025286 32742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:08.025298 32742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:08.026867 32742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.2.84:45811/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:41957
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:08.027112 32742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:08.027380 32742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:08.032346 32748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:08.032392 32750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:08.032392 32747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:08.032454 32742 server_base.cc:1034] running on GCE node
I20260927 07:06:08.032889 32742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:08.033231 32742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:08.034413 32742 hybrid_clock.cc:648] HybridClock initialized: now 1790492768034389 us; error 40 us; skew 500 ppm
Sep 27 07:06:08 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492768, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:08.042647 32742 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:06:08.043124 32742 webserver.cc:460] Webserver started at http://127.21.2.126:32821/ using document root <none> and password file <none>
I20260927 07:06:08.043457 32742 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:08.043501 32742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:08.043637 32742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:08 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492768, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:08.095721 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "f2b9918a1d7546199a7597e1c05c5fd7"
format_stamp: "Formatted at 2026-09-27 07:06:08 on dist-test-slave-ltqz"
server_key: "e5220cb44cb711f19d03cddd3de07233"
server_key_iv: "f4f95b2fc86bc691ad3d0b11bab84cca"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:08.096138 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "f2b9918a1d7546199a7597e1c05c5fd7"
format_stamp: "Formatted at 2026-09-27 07:06:08 on dist-test-slave-ltqz"
server_key: "e5220cb44cb711f19d03cddd3de07233"
server_key_iv: "f4f95b2fc86bc691ad3d0b11bab84cca"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:08.097611 32742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:06:08.112785 32757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:08.113067 32742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:08.113190 32742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "f2b9918a1d7546199a7597e1c05c5fd7"
format_stamp: "Formatted at 2026-09-27 07:06:08 on dist-test-slave-ltqz"
server_key: "e5220cb44cb711f19d03cddd3de07233"
server_key_iv: "f4f95b2fc86bc691ad3d0b11bab84cca"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:08.113294 32742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:08.156376 32742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:08.158336 32742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:08.158486 32742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:08.163096 32742 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:41957
I20260927 07:06:08.163132 341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:41957 every 8 connection(s)
I20260927 07:06:08.163548 32742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:08.164224 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:08.165172 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 32742
I20260927 07:06:08.165287 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:06:08.165495 21513 external_mini_cluster.cc:1428] Setting key cf08269e669d3bdbb729e7f717ca5819
I20260927 07:06:08.167002 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: Bootstrap starting.
I20260927 07:06:08.167802 342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:08.168146 342 log.cc:824] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:08.168828 342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: No bootstrap required, opened a new log
I20260927 07:06:08.171314 342 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } }
I20260927 07:06:08.171540 342 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:08.171583 342 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b9918a1d7546199a7597e1c05c5fd7, State: Initialized, Role: FOLLOWER
I20260927 07:06:08.171733 342 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } }
I20260927 07:06:08.171826 342 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:08.171864 342 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:08.171932 342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:08.172583 342 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } }
I20260927 07:06:08.172669 342 leader_election.cc:302] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b9918a1d7546199a7597e1c05c5fd7; no voters:
Sep 27 07:06:08 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:08.172853 342 leader_election.cc:288] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:08.172937 347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:08.173111 347 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [term 1 LEADER]: Becoming Leader. State: Replica: f2b9918a1d7546199a7597e1c05c5fd7, State: Running, Role: LEADER
I20260927 07:06:08.173167 342 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:08.173237 347 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } }
I20260927 07:06:08.173666 347 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b9918a1d7546199a7597e1c05c5fd7. Latest consensus state: current_term: 1 leader_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } } }
I20260927 07:06:08.174165 348 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b9918a1d7546199a7597e1c05c5fd7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41957 } } }
I20260927 07:06:08.174284 347 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:08.174371 348 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:08.174723 342 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:06:08.174736 361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:08.178038 361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:08.180610 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:08.166057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.21.2.126:41957)
0927 07:06:08.166243 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:08.166251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:08.166275 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:06:08.166866 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:08.166874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:08.166928 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:08.167048 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:08.167878 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:08.168998 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:08.170075 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:08.170471 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:08.170547 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:08.173529 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:08.173534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:08.173535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:08.173547 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:08.177769 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:08.178355 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:08.178359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:08.178360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:08.178542 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:08.178789 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:08.178790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:08.178790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:08.179449 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:08.179709 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:08.179806 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:06:08.180228 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260927 07:06:08.186414 361 catalog_manager.cc:1347] Generated new cluster ID: 7f61c91ad4f84ec697371a98990f441d
I20260927 07:06:08.186478 361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:08.223088 361 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:08.223902 361 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:08.228178 361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: Generated new TSK 0
I20260927 07:06:08.228457 361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:06:08.902017 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:45811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:09.004741 387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:09.004959 387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:09.004985 387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:09.006244 387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:09.006290 387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:09.006320 387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:06:09.007908 387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.2.84:45811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:09.008160 387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:09.008422 387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:09.014606 392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.014622 395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.014633 393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:09.014801 387 server_base.cc:1034] running on GCE node
I20260927 07:06:09.015126 387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:09.015338 387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:09.016523 387 hybrid_clock.cc:648] HybridClock initialized: now 1790492769016507 us; error 34 us; skew 500 ppm
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:09.019456 387 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:06:09.020030 387 webserver.cc:460] Webserver started at http://127.21.2.65:42989/ using document root <none> and password file <none>
I20260927 07:06:09.020299 387 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:09.020368 387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:09.020509 387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:09.035694 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "b06e98a796b8410aa9334642093d2a56"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "f45b0636d335343f630073ba7641eb43"
server_key_iv: "bdea4d98329316f2681a0e28761b74d1"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.036260 387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "b06e98a796b8410aa9334642093d2a56"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "f45b0636d335343f630073ba7641eb43"
server_key_iv: "bdea4d98329316f2681a0e28761b74d1"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.038254 387 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:09.059298 404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.059618 387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 07:06:09.059775 387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "b06e98a796b8410aa9334642093d2a56"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "f45b0636d335343f630073ba7641eb43"
server_key_iv: "bdea4d98329316f2681a0e28761b74d1"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.059969 387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:09.087842 387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:09.090041 387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:09.090237 387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:09.090529 387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:09.090952 387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:09.091015 387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.091060 387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:09.091092 387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.099298 387 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:36837
I20260927 07:06:09.099781 387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:09.100373 517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:36837 every 8 connection(s)
I20260927 07:06:09.101071 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 387
I20260927 07:06:09.101176 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:09.101382 21513 external_mini_cluster.cc:1428] Setting key de712c1cf91f1e15492a59905c6bc169
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:09.115281 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.100570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:41957 (local address 127.21.2.65:51127)
0927 07:06:09.101119 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.101169 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.101861 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.102288 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.102293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.102510 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:09.102715 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.102720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.103714 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.103715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.104999 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.105002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.105557 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.105558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.105740 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.106369 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:09.106380 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:09.108126 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:09.110310 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.110318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.110321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.110640 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.110870 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.110871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.110871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.111018 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.114490 (+ 3472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:09.114499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:09.114708 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.114839 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.115025 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":87,"threads_started":1}
I20260927 07:06:09.115871 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.100928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:51127 (local address 127.21.2.126:41957)
0927 07:06:09.101092 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.101095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.102027 (+ 932us) server_negotiation.cc:407] Connection header received
0927 07:06:09.102057 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.102058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.102073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.102147 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:09.102939 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.103601 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.105103 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.105408 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.105487 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.108246 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:09.108247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:09.108248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:09.108256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:09.110199 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.110730 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.110731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.110732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.110800 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.111090 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.111091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.111091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.111234 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:09.111294 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.115669 (+ 4375us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.115770 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260927 07:06:09.116699 518 heartbeater.cc:347] Connected to a master server at 127.21.2.126:41957
I20260927 07:06:09.116793 518 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:09.117007 518 heartbeater.cc:511] Master 127.21.2.126:41957 requested a full tablet report, sending...
I20260927 07:06:09.117615 306 ts_manager.cc:194] Registered new tserver with Master: b06e98a796b8410aa9334642093d2a56 (127.21.2.65:36837)
I20260927 07:06:09.118570 306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:51127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:06:09.142505 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:45811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 07:06:09.267992 526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:09.268291 526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:09.268328 526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:09.270133 526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:09.270239 526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:09.270298 526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:06:09.272642 526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.2.84:45811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:09.273051 526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:09.273377 526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:06:09.279976 526 server_base.cc:1034] running on GCE node
W20260927 07:06:09.280202 531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.280226 534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.280385 532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:09.280732 526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:09.281014 526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:09.282215 526 hybrid_clock.cc:648] HybridClock initialized: now 1790492769282187 us; error 42 us; skew 500 ppm
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:09.285503 526 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:06:09.286165 526 webserver.cc:460] Webserver started at http://127.21.2.66:40939/ using document root <none> and password file <none>
I20260927 07:06:09.286428 526 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:09.286502 526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:09.286680 526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:09.308357 526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "96e6f516361049b2978a49323ce74c10"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "4c24215de0fe99117c2922a0e27f895d"
server_key_iv: "747bdea88397f1aa2c2c78af80493510"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.308812 526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "96e6f516361049b2978a49323ce74c10"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "4c24215de0fe99117c2922a0e27f895d"
server_key_iv: "747bdea88397f1aa2c2c78af80493510"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.310998 526 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:06:09.323138 542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.323462 526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:09.323580 526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "96e6f516361049b2978a49323ce74c10"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "4c24215de0fe99117c2922a0e27f895d"
server_key_iv: "747bdea88397f1aa2c2c78af80493510"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.323689 526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:09.348479 526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:09.350808 526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:09.351019 526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:09.351300 526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:09.351730 526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:09.351827 526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.351910 526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:09.351948 526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.358693 526 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:39245
I20260927 07:06:09.359088 526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:09.360647 655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:39245 every 8 connection(s)
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:09.367271 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 526
I20260927 07:06:09.367422 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:09.367662 21513 external_mini_cluster.cc:1428] Setting key 660e0b77cad4b33b5603088ac855a377
I20260927 07:06:09.370985 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.360261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:34023 (local address 127.21.2.126:41957)
0927 07:06:09.360313 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.360316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.361195 (+ 879us) server_negotiation.cc:407] Connection header received
0927 07:06:09.361726 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.361727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.361739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.361795 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:09.362410 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.362921 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.364107 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.364327 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.364423 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.366906 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:09.366908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:09.366908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:09.366915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:09.368791 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.369596 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.369598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.369598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.369666 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.370012 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.370013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.370013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.370177 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:09.370232 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.370714 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.370845 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:06:09.372148 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.359960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:41957 (local address 127.21.2.66:34023)
0927 07:06:09.361054 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.361103 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.361618 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.361927 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.361931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.362114 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:09.362302 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.362306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.363020 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.363021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.364016 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.364019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.364433 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.364434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.364575 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.365205 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:09.365213 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:09.366518 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:09.369086 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.369095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.369098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.369493 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.369767 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.369768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.369768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.369930 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.370290 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:09.370297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:09.370528 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.371450 (+ 922us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.371685 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":82,"threads_started":1}
I20260927 07:06:09.372855 656 heartbeater.cc:347] Connected to a master server at 127.21.2.126:41957
I20260927 07:06:09.372962 656 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:09.373168 656 heartbeater.cc:511] Master 127.21.2.126:41957 requested a full tablet report, sending...
I20260927 07:06:09.373847 306 ts_manager.cc:194] Registered new tserver with Master: 96e6f516361049b2978a49323ce74c10 (127.21.2.66:39245)
I20260927 07:06:09.374419 306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:34023
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:06:09.398711 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:45811/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:09.495357 661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:09.495774 661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:09.495882 661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:09.497679 661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:09.497814 661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:09.497905 661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:06:09.500183 661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.2.84:45811/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:09.500620 661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:09.500910 661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:09.507506 667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:09.507763 661 server_base.cc:1034] running on GCE node
W20260927 07:06:09.508008 666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.508177 669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:09.508491 661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:09.508795 661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:09.509977 661 hybrid_clock.cc:648] HybridClock initialized: now 1790492769509959 us; error 38 us; skew 500 ppm
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:09.513286 661 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:06:09.513818 661 webserver.cc:460] Webserver started at http://127.21.2.67:36905/ using document root <none> and password file <none>
I20260927 07:06:09.514061 661 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:09.514111 661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:09.514236 661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:09.528429 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "ce29020251194eca8abdf16d00cdfcca"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "08f8c0ed36c76bc46889b56d089810fb"
server_key_iv: "1d06cb33ffcf36acee6ffefd6f22ae18"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.529011 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "ce29020251194eca8abdf16d00cdfcca"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "08f8c0ed36c76bc46889b56d089810fb"
server_key_iv: "1d06cb33ffcf36acee6ffefd6f22ae18"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.531172 661 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:09.544312 676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.544562 661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:06:09.544651 661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "ce29020251194eca8abdf16d00cdfcca"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-ltqz"
server_key: "08f8c0ed36c76bc46889b56d089810fb"
server_key_iv: "1d06cb33ffcf36acee6ffefd6f22ae18"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:09.544739 661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:09.562858 661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:09.565491 661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:09.565685 661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:09.565946 661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:09.566359 661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:09.566453 661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.566530 661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:09.566594 661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.574049 661 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:34317
I20260927 07:06:09.574573 661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:09.575353 789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:34317 every 8 connection(s)
I20260927 07:06:09.584651 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 661
I20260927 07:06:09.584759 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:09.584981 21513 external_mini_cluster.cc:1428] Setting key 22d2eac71ced41ee42a39f4722b23ad1
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:09.589880 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.576021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:49069 (local address 127.21.2.126:41957)
0927 07:06:09.576069 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.576072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.579656 (+ 3584us) server_negotiation.cc:407] Connection header received
0927 07:06:09.580188 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.580190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.580202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.580265 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:09.580838 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.581583 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.582761 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.582966 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.583055 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.585606 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:09.585607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:09.585608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:09.585615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:09.587238 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.587686 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.587687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.587687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.587751 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.589196 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.589197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.589197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.589320 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:09.589400 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.589699 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.589786 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:06:09.590072 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.576026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:41957 (local address 127.21.2.67:49069)
0927 07:06:09.579505 (+ 3479us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.579557 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.580082 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.580342 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.580346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.580538 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:09.580731 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.580734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.581705 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.581706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.582673 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.582676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.583068 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.583069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.583211 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.583984 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:09.583993 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:09.585503 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:09.587319 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.587324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.587327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.587605 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.588991 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:09.588992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:09.588992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:09.589105 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:09.589392 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:09.589399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:09.589559 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.589664 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.589820 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3426,"thread_start_us":81,"threads_started":1}
I20260927 07:06:09.590571 790 heartbeater.cc:347] Connected to a master server at 127.21.2.126:41957
I20260927 07:06:09.590687 790 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:09.590874 790 heartbeater.cc:511] Master 127.21.2.126:41957 requested a full tablet report, sending...
I20260927 07:06:09.591272 306 ts_manager.cc:194] Registered new tserver with Master: ce29020251194eca8abdf16d00cdfcca (127.21.2.67:34317)
I20260927 07:06:09.591694 306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:49069
I20260927 07:06:09.596676 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:09.607028 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.598159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.21.2.126:41957)
0927 07:06:09.598205 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.598207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.599218 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 07:06:09.599320 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.599321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.599331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.599382 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:09.600140 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.601020 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.602121 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.602411 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.602449 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.604033 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:09.604035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:09.604035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:09.604043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:09.605806 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.606196 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.606197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.606197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.606258 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.606464 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.606465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.606465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.606596 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:09.606702 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.606840 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.606928 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:06:09.608959 306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:09.611249 306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:09.623121 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.617903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:34317 (local address 127.0.0.1:56464)
0927 07:06:09.618773 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.618781 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.618834 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.619118 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.619121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.619123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:09.619185 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.619187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.620496 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.620497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.621828 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.621831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.622791 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.622793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.622878 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.622893 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.622984 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.623034 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":58,"threads_started":1}
I20260927 07:06:09.623134 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.618084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.21.2.67:34317)
0927 07:06:09.618728 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.618733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.618901 (+ 168us) server_negotiation.cc:407] Connection header received
0927 07:06:09.618944 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.618947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.618962 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.619012 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:09.619252 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.620398 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.621944 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.622692 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.622768 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.622850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.623008 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.623049 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":69,"threads_started":1}
I20260927 07:06:09.624475 714 tablet_service.cc:1467] Processing CreateTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 (DEFAULT_TABLE table=test-table [id=201cdf65d52f479d939e30e70be27fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:09.624890 714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3497e3fd66cf4dab89f92e90f5d9d954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:09.629079 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.617663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36837 (local address 127.0.0.1:44810)
0927 07:06:09.618477 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.618519 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.618585 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.623859 (+ 5274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.623860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.623862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:09.623905 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.623907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.626304 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.626306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.627748 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.627752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.628827 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.628829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.628896 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.628903 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.628944 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.628988 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":81,"threads_started":1}
I20260927 07:06:09.629459 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.619197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.21.2.65:36837)
0927 07:06:09.623681 (+ 4484us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.623685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.623697 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:06:09.623733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.623735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.623752 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.624114 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:09.624194 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.626172 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.627882 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.628686 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.629225 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.629305 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.629338 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.629383 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4447,"thread_start_us":60,"threads_started":1}
I20260927 07:06:09.629498 808 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Bootstrap starting.
I20260927 07:06:09.630244 452 tablet_service.cc:1467] Processing CreateTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 (DEFAULT_TABLE table=test-table [id=201cdf65d52f479d939e30e70be27fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:09.630601 452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3497e3fd66cf4dab89f92e90f5d9d954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:09.630820 808 tablet_bootstrap.cc:654] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:09.631167 808 log.cc:824] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:09.632432 808 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: No bootstrap required, opened a new log
I20260927 07:06:09.632509 808 ts_tablet_manager.cc:1397] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:06:09.632773 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.618291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:39245 (local address 127.0.0.1:40936)
0927 07:06:09.623448 (+ 5157us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.623460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.623518 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.629018 (+ 5500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.629019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.629022 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:09.629062 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.629064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.630365 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.630367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.631515 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.631517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.632554 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.632555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.632623 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.632628 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.632659 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.632698 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5135,"thread_start_us":83,"threads_started":1}
I20260927 07:06:09.633174 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.618409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.21.2.66:39245)
0927 07:06:09.623522 (+ 5113us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.623527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.628854 (+ 5327us) server_negotiation.cc:407] Connection header received
0927 07:06:09.628906 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.628908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.628925 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.629007 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:09.629147 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.630231 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.631636 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.632448 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.632941 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.633031 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.633056 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.633095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5062,"thread_start_us":74,"threads_started":1}
I20260927 07:06:09.633766 590 tablet_service.cc:1467] Processing CreateTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 (DEFAULT_TABLE table=test-table [id=201cdf65d52f479d939e30e70be27fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:09.634089 590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3497e3fd66cf4dab89f92e90f5d9d954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:09.634548 808 raft_consensus.cc:348] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.634698 808 raft_consensus.cc:374] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:09.634732 808 raft_consensus.cc:729] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce29020251194eca8abdf16d00cdfcca, State: Initialized, Role: FOLLOWER
I20260927 07:06:09.634876 808 consensus_queue.cc:260] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.635135 808 ts_tablet_manager.cc:1428] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:09.635926 790 heartbeater.cc:503] Master 127.21.2.126:41957 was elected leader, sending a full tablet report...
I20260927 07:06:09.636829 810 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: Bootstrap starting.
I20260927 07:06:09.637573 810 tablet_bootstrap.cc:654] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:09.637946 810 log.cc:824] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:09.638268 812 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Bootstrap starting.
I20260927 07:06:09.639230 812 tablet_bootstrap.cc:654] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:09.639544 812 log.cc:824] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:09.640396 810 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: No bootstrap required, opened a new log
I20260927 07:06:09.640446 812 tablet_bootstrap.cc:492] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: No bootstrap required, opened a new log
I20260927 07:06:09.640511 812 ts_tablet_manager.cc:1397] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:09.640528 810 ts_tablet_manager.cc:1397] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 07:06:09.642300 812 raft_consensus.cc:348] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.642454 812 raft_consensus.cc:374] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:09.642484 812 raft_consensus.cc:729] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e6f516361049b2978a49323ce74c10, State: Initialized, Role: FOLLOWER
I20260927 07:06:09.642717 812 consensus_queue.cc:260] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.643024 812 ts_tablet_manager.cc:1428] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:09.643170 656 heartbeater.cc:503] Master 127.21.2.126:41957 was elected leader, sending a full tablet report...
I20260927 07:06:09.644021 810 raft_consensus.cc:348] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.644208 810 raft_consensus.cc:374] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:09.644249 810 raft_consensus.cc:729] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06e98a796b8410aa9334642093d2a56, State: Initialized, Role: FOLLOWER
I20260927 07:06:09.644395 810 consensus_queue.cc:260] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.644769 810 ts_tablet_manager.cc:1428] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 07:06:09.644941 518 heartbeater.cc:503] Master 127.21.2.126:41957 was elected leader, sending a full tablet report...
Sep 27 07:06:09 dist-test-slave-ltqz krb5kdc[32298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:09.687007 811 raft_consensus.cc:482] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:09.687152 811 raft_consensus.cc:504] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.687614 811 leader_election.cc:288] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06e98a796b8410aa9334642093d2a56 (127.21.2.65:36837), 96e6f516361049b2978a49323ce74c10 (127.21.2.66:39245)
I20260927 07:06:09.691404 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.687760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36837 (local address 127.21.2.67:55909)
0927 07:06:09.688001 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.688013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.688061 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.688358 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.688358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.688360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:09.688429 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.688431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.689299 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.689300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.690399 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.690402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.691148 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.691149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.691218 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.691221 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.691265 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.691304 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 07:06:09.692194 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.688163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:55909 (local address 127.21.2.65:36837)
0927 07:06:09.688207 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.688209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.688217 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:06:09.688245 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.688246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.688259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.688308 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:09.688506 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.689193 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.690509 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.691034 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.691102 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.691113 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.692074 (+ 961us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.692126 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:06:09.692489 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497e3fd66cf4dab89f92e90f5d9d954" candidate_uuid: "ce29020251194eca8abdf16d00cdfcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06e98a796b8410aa9334642093d2a56" is_pre_election: true
I20260927 07:06:09.692654 472 raft_consensus.cc:2454] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce29020251194eca8abdf16d00cdfcca in term 0.
I20260927 07:06:09.693099 679 leader_election.cc:302] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06e98a796b8410aa9334642093d2a56, ce29020251194eca8abdf16d00cdfcca; no voters:
I20260927 07:06:09.693233 811 raft_consensus.cc:2781] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:09.693270 811 raft_consensus.cc:482] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:09.693292 811 raft_consensus.cc:3037] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:09.694224 811 raft_consensus.cc:504] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.694376 811 leader_election.cc:288] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [CANDIDATE]: Term 1 election: Requested vote from peers b06e98a796b8410aa9334642093d2a56 (127.21.2.65:36837), 96e6f516361049b2978a49323ce74c10 (127.21.2.66:39245)
I20260927 07:06:09.694737 472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497e3fd66cf4dab89f92e90f5d9d954" candidate_uuid: "ce29020251194eca8abdf16d00cdfcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06e98a796b8410aa9334642093d2a56"
I20260927 07:06:09.694818 472 raft_consensus.cc:3037] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:09.695747 472 raft_consensus.cc:2454] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce29020251194eca8abdf16d00cdfcca in term 1.
I20260927 07:06:09.696249 679 leader_election.cc:302] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06e98a796b8410aa9334642093d2a56, ce29020251194eca8abdf16d00cdfcca; no voters:
I20260927 07:06:09.697239 811 raft_consensus.cc:2781] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:09.697438 811 raft_consensus.cc:686] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 1 LEADER]: Becoming Leader. State: Replica: ce29020251194eca8abdf16d00cdfcca, State: Running, Role: LEADER
I20260927 07:06:09.697548 811 consensus_queue.cc:237] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } }
I20260927 07:06:09.698431 306 catalog_manager.cc:5495] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca reported cstate change: term changed from 0 to 1, leader changed from <none> to ce29020251194eca8abdf16d00cdfcca (127.21.2.67). New cstate: current_term: 1 leader_uuid: "ce29020251194eca8abdf16d00cdfcca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce29020251194eca8abdf16d00cdfcca" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 34317 } health_report { overall_health: HEALTHY } } }
I20260927 07:06:09.700191 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.687888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:39245 (local address 127.21.2.67:34873)
0927 07:06:09.691512 (+ 3624us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.691523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:09.691569 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:09.696276 (+ 4707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:09.696278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:09.696280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:09.696323 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.696324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.698101 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.698102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.699037 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:09.699040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.699937 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.699938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:09.700006 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.700010 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:06:09.700065 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 07:06:09.700107 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3611,"thread_start_us":123,"threads_started":1}
I20260927 07:06:09.700593 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.687953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:34873 (local address 127.21.2.66:39245)
0927 07:06:09.688004 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.688006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.691717 (+ 3711us) server_negotiation.cc:407] Connection header received
0927 07:06:09.691748 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.691749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.691762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.691810 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:09.696420 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.697092 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.699138 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.699776 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.699884 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.699896 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.700475 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.700525 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:06:09.700979 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497e3fd66cf4dab89f92e90f5d9d954" candidate_uuid: "ce29020251194eca8abdf16d00cdfcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e6f516361049b2978a49323ce74c10" is_pre_election: true
I20260927 07:06:09.701170 607 raft_consensus.cc:2454] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce29020251194eca8abdf16d00cdfcca in term 0.
I20260927 07:06:09.701314 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497e3fd66cf4dab89f92e90f5d9d954" candidate_uuid: "ce29020251194eca8abdf16d00cdfcca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e6f516361049b2978a49323ce74c10"
I20260927 07:06:09.701397 605 raft_consensus.cc:3037] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:09.702342 605 raft_consensus.cc:2454] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce29020251194eca8abdf16d00cdfcca in term 1.
I20260927 07:06:09.705345 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.701835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.21.2.67:34317)
0927 07:06:09.701877 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.701878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.701886 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:06:09.701930 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.701931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.701950 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.702015 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:09.702219 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.702880 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.704246 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.704555 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.704613 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.704800 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:09.704941 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:09.705013 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.705239 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.705256 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:06:09.707434 605 raft_consensus.cc:1261] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 1 FOLLOWER]: Refusing update from remote peer ce29020251194eca8abdf16d00cdfcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:09.707558 472 raft_consensus.cc:1261] T 3497e3fd66cf4dab89f92e90f5d9d954 P b06e98a796b8410aa9334642093d2a56 [term 1 FOLLOWER]: Refusing update from remote peer ce29020251194eca8abdf16d00cdfcca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:09.707871 818 consensus_queue.cc:1035] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e6f516361049b2978a49323ce74c10" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:09.711211 811 consensus_queue.cc:1035] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06e98a796b8410aa9334642093d2a56" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:09.714771 821 mvcc.cc:205] Tried to move back new op lower bound from 7333858384719093760 to 7333858384681861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333858384719093760}]
I20260927 07:06:09.715302 824 mvcc.cc:205] Tried to move back new op lower bound from 7333858384719093760 to 7333858384681861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333858384719093760}]
I20260927 07:06:09.717120 822 mvcc.cc:205] Tried to move back new op lower bound from 7333858384719093760 to 7333858384681861120. Current Snapshot: MvccSnapshot[applied={T|T < 7333858384719093760}]
I20260927 07:06:09.726011 305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:09.726130 305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:09.727216 305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b9918a1d7546199a7597e1c05c5fd7: Sending DeleteTablet for 3 replicas of tablet 3497e3fd66cf4dab89f92e90f5d9d954
I20260927 07:06:09.727738 452 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:09 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:44810
I20260927 07:06:09.727979 714 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:09 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:56464
I20260927 07:06:09.728341 590 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497e3fd66cf4dab89f92e90f5d9d954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:09 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:40936
I20260927 07:06:09.728992 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 387
I20260927 07:06:09.735530 832 tablet_replica.cc:330] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: stopping tablet replica
I20260927 07:06:09.735757 832 raft_consensus.cc:2229] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:09.736116 832 raft_consensus.cc:2258] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:09.736291 833 tablet_replica.cc:330] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: stopping tablet replica
I20260927 07:06:09.736459 833 raft_consensus.cc:2229] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:09.736696 833 raft_consensus.cc:2258] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:09.739810 832 ts_tablet_manager.cc:1905] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:09.740259 833 ts_tablet_manager.cc:1905] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:09.742022 832 ts_tablet_manager.cc:1918] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:09.743691 833 ts_tablet_manager.cc:1918] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:09.743829 832 log.cc:1196] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/3497e3fd66cf4dab89f92e90f5d9d954
I20260927 07:06:09.743947 833 log.cc:1196] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/3497e3fd66cf4dab89f92e90f5d9d954
I20260927 07:06:09.744304 832 ts_tablet_manager.cc:1939] T 3497e3fd66cf4dab89f92e90f5d9d954 P ce29020251194eca8abdf16d00cdfcca: Deleting consensus metadata
I20260927 07:06:09.744910 833 ts_tablet_manager.cc:1939] T 3497e3fd66cf4dab89f92e90f5d9d954 P 96e6f516361049b2978a49323ce74c10: Deleting consensus metadata
I20260927 07:06:09.745399 32759 catalog_manager.cc:4841] TS ce29020251194eca8abdf16d00cdfcca (127.21.2.67:34317): tablet 3497e3fd66cf4dab89f92e90f5d9d954 (table test-table [id=201cdf65d52f479d939e30e70be27fcb]) successfully deleted
I20260927 07:06:09.745946 32758 catalog_manager.cc:4841] TS 96e6f516361049b2978a49323ce74c10 (127.21.2.66:39245): tablet 3497e3fd66cf4dab89f92e90f5d9d954 (table test-table [id=201cdf65d52f479d939e30e70be27fcb]) successfully deleted
W20260927 07:06:09.765314 32760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:36837 (error 108)
I20260927 07:06:09.773038 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 526
I20260927 07:06:09.774374 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.774059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36837 (local address 127.0.0.1:44826)
0927 07:06:09.774267 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:09.774331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260927 07:06:09.775204 32760 catalog_manager.cc:4584] TS b06e98a796b8410aa9334642093d2a56 (127.21.2.65:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3497e3fd66cf4dab89f92e90f5d9d954: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36837: connect: Connection refused (error 111)
I20260927 07:06:09.779038 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 661
I20260927 07:06:09.784425 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 32742
I20260927 07:06:09.789223 21513 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:06:10.133494 21513 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:06:10.133594 21513 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:06:10.478771 21513 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:06:10.479 UTC [32328] LOG: received smart shutdown request
2026-09-27 07:06:10.482 UTC [32328] LOG: background worker "logical replication launcher" (PID 32337) exited with exit code 1
2026-09-27 07:06:10.482 UTC [32332] LOG: shutting down
2026-09-27 07:06:10.644 UTC [32328] LOG: database system is shut down
2026-09-27T07:06:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46291 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:10 dist-test-slave-ltqz krb5kdc[843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:10 dist-test-slave-ltqz krb5kdc[843](info): set up 2 sockets
Sep 27 07:06:10 dist-test-slave-ltqz krb5kdc[843](info): commencing operation
krb5kdc: starting...
W20260927 07:06:11.811352 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:06:11 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangeradmin_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangeradmin_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangerlookup_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangerlookup_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
I20260927 07:06:11.975911 21513 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHv0k7x/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/postgres -l logfile start
2026-09-27 07:06:14.376 UTC [873] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:06:14.376 UTC [873] LOG: listening on IPv4 address "127.21.2.84", port 36253
2026-09-27 07:06:14.379 UTC [873] LOG: listening on Unix socket "/tmp/.s.PGSQL.36253"
2026-09-27 07:06:14.414 UTC [876] LOG: database system was shut down at 2026-09-27 07:06:13 UTC
2026-09-27 07:06:14.418 UTC [873] LOG: database system is ready to accept connections
I20260927 07:06:15.367327 21513 mini_postgres.cc:96] Postgres bound to 36253
2026-09-27 07:06:15.373 UTC [884] FATAL: database "slave" does not exist
127.21.2.84:36253 - accepting connections
I20260927 07:06:15.373878 21513 mini_ranger.cc:161] Starting Ranger...
I20260927 07:06:15.388253 21513 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:06:15.823063 21513 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:06:15.823156 21513 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin
2026-09-27 07:06:16,113 [I] DB FLAVOR :POSTGRES
2026-09-27 07:06:16,114 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:06:16,114 [I] Checking connection..
2026-09-27 07:06:16,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:06:16,357 [I] Checking connection passed.
2026-09-27 07:06:16,358 [I] --------- Verifying version history table ---------
2026-09-27 07:06:16,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:06:16,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:06:16,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:06:16,829 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:06:16,830 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:06:16,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:06:17.062 UTC [948] WARNING: there is no transaction in progress
2026-09-27 07:06:17,068 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:06:17,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:06:17,331 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:06:17,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:06:17,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:06:17,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:06:17,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ltqz.c.gcp-upstream.internal','N') ;"
2026-09-27 07:06:18,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:06:18,281 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:06:18,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:06:18,524 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:06:18,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:06:18,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:06:19,012 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:06:19,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:06:19.952 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:19.968 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:19.984 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:19.998 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.014 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.034 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.098 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.105 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.119 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.128 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.138 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.152 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.163 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.169 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.176 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20.183 UTC [1056] WARNING: there is no transaction in progress
2026-09-27 07:06:20,822 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:06:20,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:06:21,061 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:06:21,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:06:21,299 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:06:21,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:06:21,561 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:06:21,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:06:21,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:06:22,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ltqz.c.gcp-upstream.internal';"
2026-09-27 07:06:22,295 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:06:22,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:06:22,539 [I] DB_PATCHES have already been applied
I20260927 07:06:22.545387 21513 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHv0k7x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:06:22.545447 21513 mini_ranger.cc:193] Using host: 127.21.2.84
I20260927 07:06:22.548444 21513 mini_ranger.cc:227] Ranger admin URL: http://127.21.2.84:41603
Sep 27, 2026 7:06:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:06:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:06:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:06:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:06:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:06:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:06:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:06:23.648854 21513 mini_ranger.cc:160] Time spent starting Ranger: real 8.275s user 0.000s sys 0.008s
Sep 27, 2026 7:06:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:06:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 7:06:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:06:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:06:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:06:44.515969 21513 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangerkms_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/rangerkms_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.2.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.2.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.2.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/HTTP_127.21.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:06:44 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:06:44.600757 21513 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:06:44.609495 21513 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:06:45.471295 21513 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:06:45.471403 21513 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms
I20260927 07:06:45.471426 21513 mini_ranger_kms.cc:227] Using postgres at 127.21.2.84:36253
2026-09-27 07:06:45,634 [I] DB FLAVOR :POSTGRES
2026-09-27 07:06:45,634 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:06:45,634 [I] Checking connection
2026-09-27 07:06:45,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:06:45,899 [I] connection success
2026-09-27 07:06:45,900 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:06:45,900 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:06:45,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:06:46,151 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:06:46,152 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:06:46,152 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:06:46,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.2.84:36253/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:06:46,443 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:06:46.913885 21513 mini_ranger_kms.cc:310] Created kms service
I20260927 07:06:47.116417 21513 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:06:47.160223 21513 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:06:47 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:47.591490 21513 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:06:47.591722 21513 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHv0k7x/build/release/bin/postgresql.jar:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHv0k7x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:06:47.591750 21513 mini_ranger_kms.cc:240] Using host: 127.21.2.84
I20260927 07:06:47.595261 21513 mini_ranger_kms.cc:276] Ranger KMS PID: 1236
I20260927 07:06:47.595356 21513 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.2.84:41101
Sep 27, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:06:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 07:06:50.054186 21513 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.453s user 0.002s sys 0.142s
I20260927 07:06:50.054287 21513 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:06:50.054332 21513 mini_ranger_kms.cc:401] 127.21.2.84:41101/kms/v1/keys
Sep 27 07:06:50 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 07:06:50 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
Sep 27 07:06:51 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492810, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.2.84@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
Sep 27 07:06:53 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:06:53.544397 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43953
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:43953
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:53.641067 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:53.641280 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:53.641304 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:53.642488 1287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:53.642534 1287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:53.642549 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:53.642561 1287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:53.642573 1287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:53.644244 1287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.2.84:41101/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:43953
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:53.644474 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:53.644677 1287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:53.649679 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:53.649672 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:53.649909 1287 server_base.cc:1034] running on GCE node
W20260927 07:06:53.650054 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:53.650307 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:53.650701 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:53.651868 1287 hybrid_clock.cc:648] HybridClock initialized: now 1790492813651848 us; error 38 us; skew 500 ppm
Sep 27 07:06:53 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:53.654529 1287 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:06:53.654963 1287 webserver.cc:460] Webserver started at http://127.21.2.126:43805/ using document root <none> and password file <none>
I20260927 07:06:53.655185 1287 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:53.655246 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:53.655441 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:53 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:53.717818 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "33c37443a65b4e4fa7541afc78369e5c"
format_stamp: "Formatted at 2026-09-27 07:06:53 on dist-test-slave-ltqz"
server_key: "3b863a98622b2f4239d49bddd4323d6c"
server_key_iv: "1d73b5f3b7b3ee4965b2dad7d4363edd"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:53.718327 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "33c37443a65b4e4fa7541afc78369e5c"
format_stamp: "Formatted at 2026-09-27 07:06:53 on dist-test-slave-ltqz"
server_key: "3b863a98622b2f4239d49bddd4323d6c"
server_key_iv: "1d73b5f3b7b3ee4965b2dad7d4363edd"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:53.720098 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:06:53.735754 1302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:53.736061 1287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:53.736179 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "33c37443a65b4e4fa7541afc78369e5c"
format_stamp: "Formatted at 2026-09-27 07:06:53 on dist-test-slave-ltqz"
server_key: "3b863a98622b2f4239d49bddd4323d6c"
server_key_iv: "1d73b5f3b7b3ee4965b2dad7d4363edd"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:53.736281 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:53.781807 1287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:53.784090 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:53.784252 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:53.789212 1287 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:43953
I20260927 07:06:53.789304 1354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:43953 every 8 connection(s)
I20260927 07:06:53.789703 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:53.790436 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:53.791879 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1287
I20260927 07:06:53.792001 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:06:53.792182 21513 external_mini_cluster.cc:1428] Setting key 11ac10b24801056813feb1f7fe181746
I20260927 07:06:53.793244 1355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Bootstrap starting.
I20260927 07:06:53.793972 1355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:53.794268 1355 log.cc:824] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:53.794946 1355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: No bootstrap required, opened a new log
I20260927 07:06:53.797223 1355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:53.797385 1355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:53.797439 1355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c37443a65b4e4fa7541afc78369e5c, State: Initialized, Role: FOLLOWER
I20260927 07:06:53.797590 1355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:53.797679 1355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:53.797717 1355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:53.797757 1355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:53.798341 1355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:53.798467 1355 leader_election.cc:302] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c37443a65b4e4fa7541afc78369e5c; no voters:
I20260927 07:06:53.798637 1355 leader_election.cc:288] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:53.798734 1360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:53.798897 1360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 LEADER]: Becoming Leader. State: Replica: 33c37443a65b4e4fa7541afc78369e5c, State: Running, Role: LEADER
I20260927 07:06:53.798926 1355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:53.799098 1360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:53.799549 1360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c37443a65b4e4fa7541afc78369e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } } }
I20260927 07:06:53.799713 1360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:53.799877 1360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c37443a65b4e4fa7541afc78369e5c. Latest consensus state: current_term: 1 leader_uuid: "33c37443a65b4e4fa7541afc78369e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } } }
I20260927 07:06:53.800009 1360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:53.800182 1369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 07:06:53 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:53.800457 1355 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:06:53.800684 1369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:53.804562 1369 catalog_manager.cc:1347] Generated new cluster ID: 4ab2e214abe5478dadbed7b818a0644c
I20260927 07:06:53.804620 1369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:53.805284 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:53.792777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.21.2.126:43953)
0927 07:06:53.792966 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:53.792974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:53.793005 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:06:53.793571 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:53.793581 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:53.793658 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:53.793816 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:53.794573 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:53.795952 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:53.797041 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:53.797695 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:53.797809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:53.801008 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:53.801011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:53.801012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:53.801023 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:53.803035 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:53.803506 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:53.803510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:53.803510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:53.803612 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:53.803853 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:53.803854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:53.803854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:53.804309 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:53.804908 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:53.804978 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 07:06:53.805096 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260927 07:06:53.839737 1369 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:53.840610 1369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:53.846308 1369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Generated new TSK 0
I20260927 07:06:53.846479 1369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:06:54.538571 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:54.657593 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:54.657855 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:54.657898 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:54.659092 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:54.659171 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:54.659219 1400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:06:54.660812 1400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:54.661084 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:54.661327 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:54.666559 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:54.666680 1400 server_base.cc:1034] running on GCE node
W20260927 07:06:54.666572 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:54.666847 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:54.667075 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:54.667443 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:54.668632 1400 hybrid_clock.cc:648] HybridClock initialized: now 1790492814668621 us; error 28 us; skew 500 ppm
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:54.671960 1400 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:06:54.672518 1400 webserver.cc:460] Webserver started at http://127.21.2.65:34683/ using document root <none> and password file <none>
I20260927 07:06:54.672817 1400 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:54.672894 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:54.673053 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:54.688068 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "06d8010d3d78411ba2fb58e7bf27d47f"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "1519d7dcc97b601c13d386b632b2721e"
server_key_iv: "2da63986a899aa1ce780f6bf9d9af617"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:54.688658 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "06d8010d3d78411ba2fb58e7bf27d47f"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "1519d7dcc97b601c13d386b632b2721e"
server_key_iv: "2da63986a899aa1ce780f6bf9d9af617"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:54.690623 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:06:54.707520 1415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.707710 1400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:06:54.707814 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "06d8010d3d78411ba2fb58e7bf27d47f"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "1519d7dcc97b601c13d386b632b2721e"
server_key_iv: "2da63986a899aa1ce780f6bf9d9af617"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:54.707926 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:54.728029 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:54.730639 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:54.730819 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:54.731168 1400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:54.731624 1400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:54.731679 1400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.731710 1400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:54.731727 1400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.743500 1400 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37865
I20260927 07:06:54.743710 1530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37865 every 8 connection(s)
I20260927 07:06:54.743952 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:54.745931 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1400
I20260927 07:06:54.746022 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:54.746208 21513 external_mini_cluster.cc:1428] Setting key 3f33fdf6e3514a3639f9ac9c18985834
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:54.759473 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:54.744760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:41093 (local address 127.21.2.126:43953)
0927 07:06:54.746055 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:54.746057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:54.746068 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:06:54.746095 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:54.746096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:54.746109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:54.747343 (+ 1234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:54.747443 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.748242 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.749874 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.750185 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.750264 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:54.753052 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:54.753054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:54.753054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:54.753063 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:54.755104 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:54.758118 (+ 3014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:54.758120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:54.758121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:54.758216 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:54.758656 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:54.758658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:54.758658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:54.758831 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:54.758907 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:54.759226 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 07:06:54.759339 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":96,"threads_started":1}
I20260927 07:06:54.759709 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:54.744730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.65:41093)
0927 07:06:54.745147 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:54.745197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:54.745854 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:54.746295 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:54.746299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:54.746939 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:54.747239 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:54.747246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.748365 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.748366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:54.749763 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:54.749766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.750424 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.750426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:54.750581 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:54.751199 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:54.751210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:54.752939 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:54.755229 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:54.755240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:54.755243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:54.755674 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:54.758343 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:54.758344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:54.758345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:54.758554 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:54.758899 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:54.758909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:54.759080 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 07:06:54.759203 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 07:06:54.759369 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":89,"threads_started":1}
I20260927 07:06:54.760224 1531 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:54.760378 1531 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:54.760615 1531 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:54.761161 1319 ts_manager.cc:194] Registered new tserver with Master: 06d8010d3d78411ba2fb58e7bf27d47f (127.21.2.65:37865)
I20260927 07:06:54.761793 1319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:41093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:06:54.778622 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:54.888263 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:54.888505 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:54.888545 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:54.889782 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:54.889856 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:54.889911 1537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:06:54.891462 1537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:54.891721 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:54.891958 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:54.897388 1544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:54.897461 1545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:54.897502 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:54.898911 1537 server_base.cc:1034] running on GCE node
I20260927 07:06:54.899160 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:54.899479 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:54.900671 1537 hybrid_clock.cc:648] HybridClock initialized: now 1790492814900660 us; error 33 us; skew 500 ppm
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:54.903405 1537 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:06:54.903962 1537 webserver.cc:460] Webserver started at http://127.21.2.66:42141/ using document root <none> and password file <none>
I20260927 07:06:54.904212 1537 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:54.904284 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:54.904433 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:54.918748 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "7c6fed14d41641ee9d15b69696022a71"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "34ed94d2e6296e940fe7400fd2945087"
server_key_iv: "f1092274f51a812553fc39830db66f8c"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:54.919517 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "7c6fed14d41641ee9d15b69696022a71"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "34ed94d2e6296e940fe7400fd2945087"
server_key_iv: "f1092274f51a812553fc39830db66f8c"
server_key_version: "kuduclusterkey@0"
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:54.922492 1537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260927 07:06:54.951553 1554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.951807 1537 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.000s
I20260927 07:06:54.951917 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "7c6fed14d41641ee9d15b69696022a71"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "34ed94d2e6296e940fe7400fd2945087"
server_key_iv: "f1092274f51a812553fc39830db66f8c"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:54.952039 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:54.976395 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:54.978662 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:54.978832 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:54.979084 1537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:54.979488 1537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:54.979542 1537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.979581 1537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:54.979606 1537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:54.986954 1537 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:39025
I20260927 07:06:54.987047 1667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:39025 every 8 connection(s)
I20260927 07:06:54.987416 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:54.990634 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1537
I20260927 07:06:54.990727 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:54.990916 21513 external_mini_cluster.cc:1428] Setting key 1ec7bef8cc0344be25cd6a25f8be7aad
Sep 27 07:06:54 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:54.998476 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:54.988234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:59627 (local address 127.21.2.126:43953)
0927 07:06:54.988314 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:54.988317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:54.988544 (+ 227us) server_negotiation.cc:407] Connection header received
0927 07:06:54.989021 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:54.989023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:54.989035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:54.989128 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:54.989716 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.990500 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.992045 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.992382 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.992459 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:54.994838 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:54.994840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:54.994840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:54.994855 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:54.996578 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:54.997149 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:54.997150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:54.997151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:54.997217 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:54.997520 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:54.997521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:54.997522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:54.997708 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:54.997793 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:54.998169 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 07:06:54.998312 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:06:54.998556 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:54.988147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.66:59627)
0927 07:06:54.988419 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:54.988451 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:54.988909 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:54.989178 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:54.989181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:54.989381 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:54.989603 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:54.989607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.990628 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.990629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:54.991946 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:54.991949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:54.992516 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:54.992518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:54.992662 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:54.993268 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:54.993278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:54.994738 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:54.996678 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:54.996685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:54.996688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:54.997044 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:54.997296 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:54.997297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:54.997297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:54.997441 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:54.997806 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:54.997813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:54.998002 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 07:06:54.998137 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 07:06:54.998315 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260927 07:06:54.999089 1668 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:54.999202 1668 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:54.999439 1668 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:55.000406 1319 ts_manager.cc:194] Registered new tserver with Master: 7c6fed14d41641ee9d15b69696022a71 (127.21.2.66:39025)
I20260927 07:06:55.000937 1319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:59627
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:06:55.027083 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:55.121191 1674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.121447 1674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.121490 1674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.122772 1674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.122844 1674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.122890 1674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:06:55.124560 1674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:55.124933 1674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.125232 1674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.130309 1680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.130510 1682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.130309 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.131284 1674 server_base.cc:1034] running on GCE node
I20260927 07:06:55.131551 1674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.131879 1674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.133082 1674 hybrid_clock.cc:648] HybridClock initialized: now 1790492815133058 us; error 40 us; skew 500 ppm
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.135773 1674 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:06:55.136299 1674 webserver.cc:460] Webserver started at http://127.21.2.67:35979/ using document root <none> and password file <none>
I20260927 07:06:55.136579 1674 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.136648 1674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:55.136783 1674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:55.150866 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "a208d135af2a44f5bb8e1ee364f34c5c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-ltqz"
server_key: "8a031b4bb463958c52d6a7195aff5202"
server_key_iv: "3beb60f0770901a6b9acbf789471dfef"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.151541 1674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "a208d135af2a44f5bb8e1ee364f34c5c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-ltqz"
server_key: "8a031b4bb463958c52d6a7195aff5202"
server_key_iv: "3beb60f0770901a6b9acbf789471dfef"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.160645 1674 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.002s
I20260927 07:06:55.178426 1689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.178692 1674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:06:55.178841 1674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "a208d135af2a44f5bb8e1ee364f34c5c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-ltqz"
server_key: "8a031b4bb463958c52d6a7195aff5202"
server_key_iv: "3beb60f0770901a6b9acbf789471dfef"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.178960 1674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:55.203595 1674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:55.205654 1674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:55.205811 1674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:55.206074 1674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:55.206411 1674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:55.206468 1674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.206511 1674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:55.206537 1674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.214295 1674 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:37369
I20260927 07:06:55.214936 1674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:55.222493 1802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:37369 every 8 connection(s)
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:55.223805 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1674
I20260927 07:06:55.223902 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:55.224087 21513 external_mini_cluster.cc:1428] Setting key a02931619e49bfa678fc8d3370d57828
I20260927 07:06:55.228860 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.216255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35849 (local address 127.21.2.126:43953)
0927 07:06:55.216343 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:55.216346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:55.216908 (+ 562us) server_negotiation.cc:407] Connection header received
0927 07:06:55.217725 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:55.217728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:55.217741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:55.217834 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:55.218711 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.219525 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.221177 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.221378 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.221436 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.224768 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:55.224770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:55.224770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:55.224779 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:55.226572 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.227087 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.227088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.227088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.227153 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.228013 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.228014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.228015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.228169 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:55.228217 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:55.228654 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 07:06:55.228751 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:06:55.229473 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.216264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.67:35849)
0927 07:06:55.216709 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:55.216772 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:55.217583 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:55.217939 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:55.217945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:55.218259 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:55.218607 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.218615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.219642 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.219644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.221075 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.221078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.221479 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.221480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.221653 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.222219 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:55.222229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:55.224648 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:55.226676 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.226683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.226685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.226990 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.227246 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.227247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.227247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.227926 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.228286 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:55.228293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:55.228483 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 07:06:55.229010 (+ 527us) client_negotiation.cc:240] Negotiation successful
0927 07:06:55.229192 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":91,"threads_started":1}
I20260927 07:06:55.229979 1803 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:55.230075 1803 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.230296 1803 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:55.230674 1319 ts_manager.cc:194] Registered new tserver with Master: a208d135af2a44f5bb8e1ee364f34c5c (127.21.2.67:37369)
I20260927 07:06:55.231168 1319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:35849
I20260927 07:06:55.238765 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:55.239506 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1400
I20260927 07:06:55.246012 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1537
I20260927 07:06:55.252055 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1674
I20260927 07:06:55.259418 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1287
I20260927 07:06:55.266390 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43953
--webserver_interface=127.21.2.126
--webserver_port=43805
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:43953
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:55.348927 1806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.349192 1806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.349239 1806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.350453 1806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:55.350534 1806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.350559 1806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.350589 1806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:55.350631 1806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:55.352406 1806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:43953
--ranger_config_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:55.352686 1806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.352929 1806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.357928 1812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.357936 1814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.357937 1811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.358325 1806 server_base.cc:1034] running on GCE node
I20260927 07:06:55.358587 1806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.358886 1806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.360098 1806 hybrid_clock.cc:648] HybridClock initialized: now 1790492815360092 us; error 37 us; skew 500 ppm
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.362720 1806 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:06:55.363171 1806 webserver.cc:460] Webserver started at http://127.21.2.126:43805/ using document root <none> and password file <none>
I20260927 07:06:55.363425 1806 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.363482 1806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:55.410526 1806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:55.411445 1822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.411635 1806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:55.411742 1806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "33c37443a65b4e4fa7541afc78369e5c"
format_stamp: "Formatted at 2026-09-27 07:06:53 on dist-test-slave-ltqz"
server_key: "3b863a98622b2f4239d49bddd4323d6c"
server_key_iv: "1d73b5f3b7b3ee4965b2dad7d4363edd"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.412173 1806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:55.449834 1806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:55.452240 1806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:55.452430 1806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:55.457039 1806 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:43953
I20260927 07:06:55.457078 1874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:43953 every 8 connection(s)
I20260927 07:06:55.457499 1806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:55.458747 1875 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:06:55.459337 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Bootstrap starting.
I20260927 07:06:55.462636 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1806
I20260927 07:06:55.462975 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:37865
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:43953
--webserver_port=34683
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:06:55.467629 1875 log.cc:824] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:55.469388 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:06:55.469727 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Bootstrap complete.
I20260927 07:06:55.472213 1875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:55.472476 1875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c37443a65b4e4fa7541afc78369e5c, State: Initialized, Role: FOLLOWER
I20260927 07:06:55.472661 1875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:55.472739 1875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:55.472793 1875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:55.472849 1875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:06:55.473832 1875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:55.473953 1875 leader_election.cc:302] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c37443a65b4e4fa7541afc78369e5c; no voters:
I20260927 07:06:55.474171 1875 leader_election.cc:288] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:06:55.474211 1879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:06:55.474326 1879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [term 2 LEADER]: Becoming Leader. State: Replica: 33c37443a65b4e4fa7541afc78369e5c, State: Running, Role: LEADER
I20260927 07:06:55.474407 1879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } }
I20260927 07:06:55.474440 1875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:55.474664 1880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33c37443a65b4e4fa7541afc78369e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } } }
I20260927 07:06:55.474763 1880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:55.475018 1881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c37443a65b4e4fa7541afc78369e5c. Latest consensus state: current_term: 2 leader_uuid: "33c37443a65b4e4fa7541afc78369e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c37443a65b4e4fa7541afc78369e5c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43953 } } }
I20260927 07:06:55.475121 1881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:55.475231 1883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:55.475775 1883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:55.475920 1883 catalog_manager.cc:1259] Loaded cluster ID: 4ab2e214abe5478dadbed7b818a0644c
I20260927 07:06:55.475965 1883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:55.476131 1875 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:06:55.479632 1883 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:55.479995 1883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Loaded TSK: 0
I20260927 07:06:55.480247 1883 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:06:55.578135 1877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.578370 1877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.578449 1877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.579813 1877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.579941 1877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.580034 1877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:06:55.582335 1877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:37865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=34683
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:55.582878 1877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.583228 1877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.590463 1915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.590852 1912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.591486 1877 server_base.cc:1034] running on GCE node
W20260927 07:06:55.591496 1913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.591827 1877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.592113 1877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.595444 1877 hybrid_clock.cc:648] HybridClock initialized: now 1790492815595394 us; error 70 us; skew 500 ppm
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.598826 1877 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:06:55.599452 1877 webserver.cc:460] Webserver started at http://127.21.2.65:34683/ using document root <none> and password file <none>
I20260927 07:06:55.599740 1877 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.599820 1877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:55.625363 1877 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 07:06:55.626180 1923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.626340 1877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:55.626418 1877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "06d8010d3d78411ba2fb58e7bf27d47f"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "1519d7dcc97b601c13d386b632b2721e"
server_key_iv: "2da63986a899aa1ce780f6bf9d9af617"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.626865 1877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:55.650605 1877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:55.652889 1877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:55.653160 1877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:55.653445 1877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:55.653815 1877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:55.653872 1877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.653915 1877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:55.653944 1877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.660619 1877 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37865
I20260927 07:06:55.660778 2036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37865 every 8 connection(s)
I20260927 07:06:55.661092 1877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:55.669056 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 1877
I20260927 07:06:55.669370 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:39025
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:43953
--webserver_port=42141
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:55.689725 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.661993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.65:42937)
0927 07:06:55.662680 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:55.662732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:55.663405 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:55.664117 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:55.664122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:55.664345 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:55.664586 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.664590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.665931 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.665932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.666987 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.666990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.673537 (+ 6547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.673538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.673714 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.683453 (+ 9739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:55.683469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:55.685164 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:55.687651 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.687662 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.687665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.687993 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.688336 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.688337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.688337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.688465 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.688971 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:55.688978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:55.689159 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 07:06:55.689291 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 07:06:55.689476 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":90,"threads_started":1}
I20260927 07:06:55.699915 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.662108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:42937 (local address 127.21.2.126:43953)
0927 07:06:55.663628 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:55.663637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:55.663677 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:06:55.663835 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:55.663842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:55.663899 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:55.664044 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:55.664711 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.665823 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.667090 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.670972 (+ 3882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.673391 (+ 2419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.685347 (+ 11956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:55.685355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:55.685355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:55.685369 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:55.687527 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.688094 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.688098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.688098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.688264 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.688531 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.688532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.688533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.688868 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:55.688943 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:55.690406 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0927 07:06:55.690542 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1465,"thread_start_us":84,"threads_started":1}
I20260927 07:06:55.701169 2037 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:55.701308 2037 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.701546 2037 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:55.702405 2037 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:43953 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:06:55.702487 2037 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.702553 2037 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:55.702953 2037 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.703023 2037 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:55.703639 2037 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:06:55.779605 2041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.779841 2041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.779882 2041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.781051 2041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.781121 2041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.781167 2041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:06:55.783005 2041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:39025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=42141
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:55.783245 2041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.783555 2041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.789896 2049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.789958 2047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.790207 2046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.790393 2041 server_base.cc:1034] running on GCE node
I20260927 07:06:55.790625 2041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.790910 2041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.792114 2041 hybrid_clock.cc:648] HybridClock initialized: now 1790492815792079 us; error 48 us; skew 500 ppm
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.795562 2041 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:06:55.796131 2041 webserver.cc:460] Webserver started at http://127.21.2.66:42141/ using document root <none> and password file <none>
I20260927 07:06:55.796413 2041 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.796504 2041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:55.812762 2041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:55.813468 2057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.813661 2041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:55.813763 2041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "7c6fed14d41641ee9d15b69696022a71"
format_stamp: "Formatted at 2026-09-27 07:06:54 on dist-test-slave-ltqz"
server_key: "34ed94d2e6296e940fe7400fd2945087"
server_key_iv: "f1092274f51a812553fc39830db66f8c"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:55.814128 2041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:55.837344 2041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:55.839879 2041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:55.840077 2041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:55.840360 2041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:55.840725 2041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:55.840778 2041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.840811 2041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:55.840830 2041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.848487 2041 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:39025
I20260927 07:06:55.849045 2041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:55.849709 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:39025 every 8 connection(s)
I20260927 07:06:55.855338 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2041
I20260927 07:06:55.855638 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.2.84:41101/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:37369
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:43953
--webserver_port=35979
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:55.861371 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.849862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.66:45321)
0927 07:06:55.850028 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:55.850069 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:55.850706 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:55.851084 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:55.851089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:55.851344 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:55.851661 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.851668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.852679 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.852680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.853960 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:55.853963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.854477 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.854478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:55.854575 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.855184 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:55.855195 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:55.857855 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:55.859732 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.859739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.859742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.860046 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.860265 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:55.860265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:55.860266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:55.860399 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:55.860717 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:55.860723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:55.860904 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 07:06:55.861018 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:06:55.861201 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260927 07:06:55.861817 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.850170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:45321 (local address 127.21.2.126:43953)
0927 07:06:55.850218 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:55.850221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:55.850231 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:06:55.850844 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:55.850846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:55.850861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:55.850896 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:55.851777 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.852557 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.854064 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.854368 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.854458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.857967 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:55.857969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:55.857970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:55.857979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:55.859631 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.860136 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.860138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.860138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.860199 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.860476 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.860477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.860477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.860627 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:55.860712 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:55.861664 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 07:06:55.861755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:06:55.862188 2171 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:55.862282 2171 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.862473 2171 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:55.863162 2171 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:43953 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:06:55.863247 2171 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.863322 2171 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:55.863633 2171 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:55.863710 2171 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:55.863974 2171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:06:55.979130 2174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.979535 2174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.979615 2174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.981544 2174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.981638 2174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.981706 2174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:06:55.983935 2174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.2.84:41101/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:37369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=35979
--tserver_master_addrs=127.21.2.126:43953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:55.984180 2174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.984534 2174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.991063 2181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.991070 2180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.991333 2183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.991420 2174 server_base.cc:1034] running on GCE node
I20260927 07:06:55.991648 2174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.991905 2174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.993103 2174 hybrid_clock.cc:648] HybridClock initialized: now 1790492815993077 us; error 36 us; skew 500 ppm
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.996251 2174 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:06:55.996769 2174 webserver.cc:460] Webserver started at http://127.21.2.67:35979/ using document root <none> and password file <none>
I20260927 07:06:55.997006 2174 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.997090 2174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:06:55 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:56.011210 2174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:06:56.012800 2191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.013021 2174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:56.013123 2174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "a208d135af2a44f5bb8e1ee364f34c5c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-ltqz"
server_key: "8a031b4bb463958c52d6a7195aff5202"
server_key_iv: "3beb60f0770901a6b9acbf789471dfef"
server_key_version: "kuduclusterkey@0"
I20260927 07:06:56.013563 2174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:56.031286 2174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:56.033828 2174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:56.034058 2174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:56.034404 2174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:56.034777 2174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:56.034832 2174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.034876 2174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:56.034904 2174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.042152 2174 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:37369
I20260927 07:06:56.042209 2304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:37369 every 8 connection(s)
I20260927 07:06:56.042584 2174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:56.044207 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2174
Sep 27 07:06:56 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:56.052569 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.043351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:50899 (local address 127.21.2.126:43953)
0927 07:06:56.043442 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:56.043444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:56.043689 (+ 245us) server_negotiation.cc:407] Connection header received
0927 07:06:56.044261 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:56.044263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:56.044276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:56.044308 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:56.045145 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.045638 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.046984 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.047200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.047275 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.049428 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:56.049429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:56.049430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:56.049438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:56.050898 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.051350 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.051351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.051352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.051450 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.051773 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.051774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.051775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.051924 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:56.051973 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:56.052370 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 07:06:56.052463 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:06:56.053226 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.043351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43953 (local address 127.21.2.67:50899)
0927 07:06:56.043535 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:56.043576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:56.044117 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:56.044493 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:56.044497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:56.044757 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:56.045046 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.045054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.045727 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.045727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.046899 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.046901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.047307 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.047308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.047479 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.048032 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:56.048041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:56.049303 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:56.051003 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.051011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.051014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.051256 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.051530 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.051531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.051532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.051670 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.052027 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:56.052033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:56.052211 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 07:06:56.052897 (+ 686us) client_negotiation.cc:240] Negotiation successful
0927 07:06:56.053057 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260927 07:06:56.053853 2305 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43953
I20260927 07:06:56.053954 2305 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.054147 2305 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:56.055244 2305 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:43953 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:06:56.055325 2305 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.055481 2305 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:56.055946 2305 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.056031 2305 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
W20260927 07:06:56.056355 2305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:06:56.058768 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.044653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.21.2.126:43953)
0927 07:06:56.044774 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:56.044776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:56.044953 (+ 177us) server_negotiation.cc:407] Connection header received
0927 07:06:56.045018 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:56.045019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:56.045031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:56.045111 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:56.045859 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.046591 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.053473 (+ 6882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.053681 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.053772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.055867 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:56.055868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:56.055868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:56.055875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:56.057607 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.057946 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.057947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.057948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.058013 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.058195 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.058196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.058196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.058337 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:56.058521 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:56.058565 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 07:06:56.058653 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
Sep 27 07:06:56 dist-test-slave-ltqz krb5kdc[843](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:56.703804 2037 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.703974 2037 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:56.704401 1839 ts_manager.cc:194] Registered new tserver with Master: 06d8010d3d78411ba2fb58e7bf27d47f (127.21.2.65:37865)
I20260927 07:06:56.705428 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:42937
I20260927 07:06:56.864190 2171 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.864362 2171 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:56.864727 1839 ts_manager.cc:194] Registered new tserver with Master: 7c6fed14d41641ee9d15b69696022a71 (127.21.2.66:39025)
I20260927 07:06:56.865304 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:45321
Sep 27 07:06:56 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
I20260927 07:06:57.056610 2305 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:57.056790 2305 heartbeater.cc:511] Master 127.21.2.126:43953 requested a full tablet report, sending...
I20260927 07:06:57.057103 1839 ts_manager.cc:194] Registered new tserver with Master: a208d135af2a44f5bb8e1ee364f34c5c (127.21.2.67:37369)
I20260927 07:06:57.057515 1839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:50899
I20260927 07:06:57.065074 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:57.078987 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.066409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.21.2.126:43953)
0927 07:06:57.066537 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.066540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.066623 (+ 83us) server_negotiation.cc:407] Connection header received
0927 07:06:57.066697 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.066698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.066710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.066786 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.067567 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.068360 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.069431 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.069753 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.069823 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.075544 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.075546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.075546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.075554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.077444 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.077870 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.077871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.077871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.077936 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.078167 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.078168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.078169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.078315 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.078415 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.078787 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.078876 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260927 07:06:57.081651 1839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:57.083956 1839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:57.093463 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.087299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:39025 (local address 127.0.0.1:44392)
0927 07:06:57.088351 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.088393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.088455 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.088802 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.088805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.088807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:57.088875 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.088877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.090259 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.090260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.091670 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.091674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.093147 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.093148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.093242 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.093256 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.093319 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.093371 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":103,"threads_started":1}
I20260927 07:06:57.093924 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.087845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.21.2.66:39025)
0927 07:06:57.087997 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.088003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.088530 (+ 527us) server_negotiation.cc:407] Connection header received
0927 07:06:57.088625 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.088627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.088643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.088707 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:57.088956 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.090074 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.091777 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.092963 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.093033 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.093107 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.093797 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.093846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260927 07:06:57.096101 2087 tablet_service.cc:1467] Processing CreateTablet for tablet 556ad1a81d3348f09c80a6e808adde3a (DEFAULT_TABLE table=test-table [id=cd5f629cd4654b42af93aae8257989c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:57.096498 2087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556ad1a81d3348f09c80a6e808adde3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:57.097210 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.087549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:37369 (local address 127.0.0.1:48580)
0927 07:06:57.091597 (+ 4048us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.091609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.091671 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.091962 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.091963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.091966 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:57.092023 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.092025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.093321 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.093323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.095208 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.095211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.096946 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.096947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.097019 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.097025 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.097075 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.097114 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4030,"thread_start_us":60,"threads_started":1}
I20260927 07:06:57.097512 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.087674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.21.2.67:37369)
0927 07:06:57.091515 (+ 3841us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.091520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.091738 (+ 218us) server_negotiation.cc:407] Connection header received
0927 07:06:57.091778 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.091781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.091797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.091868 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:57.092105 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.093216 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.095968 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.096756 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.096840 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.096911 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.097387 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.097437 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3803,"thread_start_us":65,"threads_started":1}
I20260927 07:06:57.097581 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.088057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.21.2.65:37865)
0927 07:06:57.088191 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.088195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.093599 (+ 5404us) server_negotiation.cc:407] Connection header received
0927 07:06:57.093718 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.093720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.093738 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.093792 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:57.094021 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.095087 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.096381 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.097096 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.097175 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.097241 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.097459 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.097502 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":48,"threads_started":1}
I20260927 07:06:57.097800 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.087705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37865 (local address 127.0.0.1:42320)
0927 07:06:57.093519 (+ 5814us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.093530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.093631 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.093907 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.093908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.093910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:57.093946 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.093947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.095207 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.095208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.096275 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.096278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.097305 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.097305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.097369 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.097372 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.097680 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.097726 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":58,"threads_started":1}
I20260927 07:06:57.098345 2239 tablet_service.cc:1467] Processing CreateTablet for tablet 556ad1a81d3348f09c80a6e808adde3a (DEFAULT_TABLE table=test-table [id=cd5f629cd4654b42af93aae8257989c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:57.098402 1971 tablet_service.cc:1467] Processing CreateTablet for tablet 556ad1a81d3348f09c80a6e808adde3a (DEFAULT_TABLE table=test-table [id=cd5f629cd4654b42af93aae8257989c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:57.098649 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556ad1a81d3348f09c80a6e808adde3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:57.098683 2239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556ad1a81d3348f09c80a6e808adde3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:57.100095 2338 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Bootstrap starting.
I20260927 07:06:57.100780 2338 tablet_bootstrap.cc:654] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:57.101060 2338 log.cc:824] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:57.101938 2338 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: No bootstrap required, opened a new log
I20260927 07:06:57.102005 2338 ts_tablet_manager.cc:1397] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:57.102243 2340 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Bootstrap starting.
I20260927 07:06:57.102851 2340 tablet_bootstrap.cc:654] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:57.103112 2340 log.cc:824] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:57.103740 2341 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Bootstrap starting.
I20260927 07:06:57.104364 2341 tablet_bootstrap.cc:654] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:57.104698 2341 log.cc:824] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:57.104863 2338 raft_consensus.cc:348] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.105154 2338 raft_consensus.cc:374] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:57.105260 2338 raft_consensus.cc:729] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6fed14d41641ee9d15b69696022a71, State: Initialized, Role: FOLLOWER
I20260927 07:06:57.105458 2340 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: No bootstrap required, opened a new log
I20260927 07:06:57.105527 2340 ts_tablet_manager.cc:1397] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 07:06:57.105484 2338 consensus_queue.cc:260] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.105968 2338 ts_tablet_manager.cc:1428] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 07:06:57.106138 2171 heartbeater.cc:503] Master 127.21.2.126:43953 was elected leader, sending a full tablet report...
I20260927 07:06:57.107968 2340 raft_consensus.cc:348] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.108124 2340 raft_consensus.cc:374] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:57.108152 2340 raft_consensus.cc:729] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a208d135af2a44f5bb8e1ee364f34c5c, State: Initialized, Role: FOLLOWER
I20260927 07:06:57.108284 2340 consensus_queue.cc:260] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.108582 2340 ts_tablet_manager.cc:1428] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:06:57.108726 2305 heartbeater.cc:503] Master 127.21.2.126:43953 was elected leader, sending a full tablet report...
I20260927 07:06:57.109575 2341 tablet_bootstrap.cc:492] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: No bootstrap required, opened a new log
I20260927 07:06:57.109652 2341 ts_tablet_manager.cc:1397] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 07:06:57.111850 2341 raft_consensus.cc:348] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.112003 2341 raft_consensus.cc:374] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:57.112031 2341 raft_consensus.cc:729] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d8010d3d78411ba2fb58e7bf27d47f, State: Initialized, Role: FOLLOWER
I20260927 07:06:57.112200 2341 consensus_queue.cc:260] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.112437 2341 ts_tablet_manager.cc:1428] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:57.112576 2037 heartbeater.cc:503] Master 127.21.2.126:43953 was elected leader, sending a full tablet report...
W20260927 07:06:57.162506 2038 tablet.cc:2369] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:06:57 dist-test-slave-ltqz krb5kdc[843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for HTTP/127.21.2.84@KRBTEST.COM
W20260927 07:06:57.294056 2306 tablet.cc:2369] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:57.350605 2172 tablet.cc:2369] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:06:57.382326 2345 raft_consensus.cc:482] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:57.382483 2345 raft_consensus.cc:504] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.382891 2345 leader_election.cc:288] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6fed14d41641ee9d15b69696022a71 (127.21.2.66:39025), 06d8010d3d78411ba2fb58e7bf27d47f (127.21.2.65:37865)
I20260927 07:06:57.392937 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.383846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:44769 (local address 127.21.2.65:37865)
0927 07:06:57.383930 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.383932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.389556 (+ 5624us) server_negotiation.cc:407] Connection header received
0927 07:06:57.389667 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.389668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.389681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.389709 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:57.389900 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.390560 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.391932 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.392506 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.392560 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.392573 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.392810 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.392858 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:06:57.393532 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.383474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37865 (local address 127.21.2.67:44769)
0927 07:06:57.389479 (+ 6005us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.389492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.389586 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.389770 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.389771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.389772 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:57.389835 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.389836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.390692 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.390693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.391817 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.391820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.392640 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.392641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.392717 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.392721 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.393374 (+ 653us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.393425 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5983,"thread_start_us":57,"threads_started":1}
I20260927 07:06:57.394030 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556ad1a81d3348f09c80a6e808adde3a" candidate_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" is_pre_election: true
I20260927 07:06:57.394227 1991 raft_consensus.cc:2454] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a208d135af2a44f5bb8e1ee364f34c5c in term 0.
I20260927 07:06:57.394611 2193 leader_election.cc:302] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d8010d3d78411ba2fb58e7bf27d47f, a208d135af2a44f5bb8e1ee364f34c5c; no voters:
I20260927 07:06:57.394759 2345 raft_consensus.cc:2781] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:57.394805 2345 raft_consensus.cc:482] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:57.394826 2345 raft_consensus.cc:3037] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:57.395787 2345 raft_consensus.cc:504] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.395937 2345 leader_election.cc:288] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [CANDIDATE]: Term 1 election: Requested vote from peers 7c6fed14d41641ee9d15b69696022a71 (127.21.2.66:39025), 06d8010d3d78411ba2fb58e7bf27d47f (127.21.2.65:37865)
I20260927 07:06:57.396296 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556ad1a81d3348f09c80a6e808adde3a" candidate_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d8010d3d78411ba2fb58e7bf27d47f"
I20260927 07:06:57.396381 1991 raft_consensus.cc:3037] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:57.397270 1991 raft_consensus.cc:2454] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a208d135af2a44f5bb8e1ee364f34c5c in term 1.
I20260927 07:06:57.397503 2193 leader_election.cc:302] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d8010d3d78411ba2fb58e7bf27d47f, a208d135af2a44f5bb8e1ee364f34c5c; no voters:
I20260927 07:06:57.397611 2345 raft_consensus.cc:2781] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:57.397815 2345 raft_consensus.cc:686] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 1 LEADER]: Becoming Leader. State: Replica: a208d135af2a44f5bb8e1ee364f34c5c, State: Running, Role: LEADER
I20260927 07:06:57.397919 2345 consensus_queue.cc:237] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } }
I20260927 07:06:57.398746 1839 catalog_manager.cc:5495] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c reported cstate change: term changed from 0 to 1, leader changed from <none> to a208d135af2a44f5bb8e1ee364f34c5c (127.21.2.67). New cstate: current_term: 1 leader_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:57.400583 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.383794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:35267 (local address 127.21.2.66:39025)
0927 07:06:57.383894 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.383896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.387593 (+ 3697us) server_negotiation.cc:407] Connection header received
0927 07:06:57.387724 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.387725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.387739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.387771 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:57.388040 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.388838 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.399462 (+ 10624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.400105 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.400164 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.400178 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.400444 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.400495 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:06:57.400823 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.383210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:39025 (local address 127.21.2.67:35267)
0927 07:06:57.387491 (+ 4281us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.387508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.387629 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.387847 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.387848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.387850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:57.387953 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.387955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.388963 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.388964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.390268 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.390270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.400271 (+ 10001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.400272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.400351 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.400356 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.400691 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.400743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4255,"thread_start_us":97,"threads_started":1}
I20260927 07:06:57.401214 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556ad1a81d3348f09c80a6e808adde3a" candidate_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6fed14d41641ee9d15b69696022a71" is_pre_election: true
I20260927 07:06:57.401371 2125 raft_consensus.cc:2454] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a208d135af2a44f5bb8e1ee364f34c5c in term 0.
I20260927 07:06:57.401212 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556ad1a81d3348f09c80a6e808adde3a" candidate_uuid: "a208d135af2a44f5bb8e1ee364f34c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6fed14d41641ee9d15b69696022a71"
I20260927 07:06:57.401605 2124 raft_consensus.cc:3037] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:57.402472 2124 raft_consensus.cc:2454] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a208d135af2a44f5bb8e1ee364f34c5c in term 1.
I20260927 07:06:57.444492 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.441221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.21.2.67:37369)
0927 07:06:57.441270 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.441272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.441282 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:06:57.441324 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.441325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.441350 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.441438 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:57.441589 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.442312 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.443480 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.443797 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.443866 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.444076 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:57.444211 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:57.444303 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.444380 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.444410 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:06:57.446794 1991 raft_consensus.cc:1261] T 556ad1a81d3348f09c80a6e808adde3a P 06d8010d3d78411ba2fb58e7bf27d47f [term 1 FOLLOWER]: Refusing update from remote peer a208d135af2a44f5bb8e1ee364f34c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:57.447196 2345 consensus_queue.cc:1035] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d8010d3d78411ba2fb58e7bf27d47f" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:57.448853 2124 raft_consensus.cc:1261] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 1 FOLLOWER]: Refusing update from remote peer a208d135af2a44f5bb8e1ee364f34c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:57.449177 2345 consensus_queue.cc:1035] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6fed14d41641ee9d15b69696022a71" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:57.451897 2354 mvcc.cc:205] Tried to move back new op lower bound from 7333858580259086336 to 7333858580062535680. Current Snapshot: MvccSnapshot[applied={T|T < 7333858580259086336}]
I20260927 07:06:57.455317 2357 mvcc.cc:205] Tried to move back new op lower bound from 7333858580259086336 to 7333858580062535680. Current Snapshot: MvccSnapshot[applied={T|T < 7333858580259086336 or (T in {7333858580259086336})}]
I20260927 07:06:57.459527 1839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:57.459651 1839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:57.460412 1839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c37443a65b4e4fa7541afc78369e5c: Sending DeleteTablet for 3 replicas of tablet 556ad1a81d3348f09c80a6e808adde3a
I20260927 07:06:57.460852 2087 tablet_service.cc:1514] Processing DeleteTablet for tablet 556ad1a81d3348f09c80a6e808adde3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:57 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:44392
I20260927 07:06:57.461002 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1877
I20260927 07:06:57.461063 2239 tablet_service.cc:1514] Processing DeleteTablet for tablet 556ad1a81d3348f09c80a6e808adde3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:57 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48580
I20260927 07:06:57.467469 2364 tablet_replica.cc:330] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71: stopping tablet replica
I20260927 07:06:57.467638 2365 tablet_replica.cc:330] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: stopping tablet replica
I20260927 07:06:57.467787 2365 raft_consensus.cc:2229] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:57.467949 2364 raft_consensus.cc:2229] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:57.468153 2365 raft_consensus.cc:2258] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:57.468322 2364 raft_consensus.cc:2258] T 556ad1a81d3348f09c80a6e808adde3a P 7c6fed14d41641ee9d15b69696022a71 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:06:57.471381 1824 connection.cc:535] client connection to 127.21.2.65:37865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 07:06:57.471488 1824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:06:57.472066 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2041
I20260927 07:06:57.473776 2365 ts_tablet_manager.cc:1905] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:57.479653 2365 ts_tablet_manager.cc:1918] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:57.479846 2365 log.cc:1196] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/556ad1a81d3348f09c80a6e808adde3a
I20260927 07:06:57.480255 2365 ts_tablet_manager.cc:1939] T 556ad1a81d3348f09c80a6e808adde3a P a208d135af2a44f5bb8e1ee364f34c5c: Deleting consensus metadata
I20260927 07:06:57.480827 1823 catalog_manager.cc:4841] TS a208d135af2a44f5bb8e1ee364f34c5c (127.21.2.67:37369): tablet 556ad1a81d3348f09c80a6e808adde3a (table test-table [id=cd5f629cd4654b42af93aae8257989c6]) successfully deleted
I20260927 07:06:57.481421 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.481263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37865 (local address 127.0.0.1:42334)
0927 07:06:57.481325 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.481384 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:06:57.481643 1824 catalog_manager.cc:4584] TS 06d8010d3d78411ba2fb58e7bf27d47f (127.21.2.65:37865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 556ad1a81d3348f09c80a6e808adde3a: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37865: connect: Connection refused (error 111)
W20260927 07:06:57.482159 2332 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:57.481547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:39025 (local address 127.0.0.1:44402)
0927 07:06:57.481846 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.481858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.481906 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.482115 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.66:39025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280}
I20260927 07:06:57.482672 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2174
I20260927 07:06:57.489931 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 1806
I20260927 07:06:57.497143 21513 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:06:57.842150 21513 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:06:57.842245 21513 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:06:58.186473 21513 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:06:58.186 UTC [873] LOG: received smart shutdown request
2026-09-27 07:06:58.191 UTC [873] LOG: background worker "logical replication launcher" (PID 882) exited with exit code 1
2026-09-27 07:06:58.191 UTC [877] LOG: shutting down
2026-09-27 07:06:58.355 UTC [873] LOG: database system is shut down
2026-09-27T07:06:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47709 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:58 dist-test-slave-ltqz krb5kdc[2375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:58 dist-test-slave-ltqz krb5kdc[2375](info): set up 2 sockets
Sep 27 07:06:58 dist-test-slave-ltqz krb5kdc[2375](info): commencing operation
krb5kdc: starting...
W20260927 07:06:59.515450 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:06:59.634853 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45181
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:59.716840 2388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:59.717057 2388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:59.717080 2388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:59.718169 2388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:59.718207 2388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:59.718230 2388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:59.718242 2388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:59.718254 2388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:59.719894 2388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:59.720104 2388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:59.720278 2388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:59.725056 2394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.725039 2396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.725039 2393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:59.725198 2388 server_base.cc:1034] running on GCE node
I20260927 07:06:59.725461 2388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:59.725765 2388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:59.726907 2388 hybrid_clock.cc:648] HybridClock initialized: now 1790492819726891 us; error 32 us; skew 500 ppm
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:59.729347 2388 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:06:59.729799 2388 webserver.cc:460] Webserver started at http://127.21.2.126:40041/ using document root <none> and password file <none>
I20260927 07:06:59.729979 2388 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:59.730027 2388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:59.730114 2388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:59.731120 2388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "8d1c390f9db94fdc8e10c845f5e5c182"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "27ae85985c3b7fbfa4a038a50dbe8ff0"
server_key_iv: "c4f0cc9604e5e768c926c56a102a7f7f"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.731482 2388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "8d1c390f9db94fdc8e10c845f5e5c182"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "27ae85985c3b7fbfa4a038a50dbe8ff0"
server_key_iv: "c4f0cc9604e5e768c926c56a102a7f7f"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.732774 2388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:59.733464 2403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:59.733668 2388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:59.733757 2388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "8d1c390f9db94fdc8e10c845f5e5c182"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "27ae85985c3b7fbfa4a038a50dbe8ff0"
server_key_iv: "c4f0cc9604e5e768c926c56a102a7f7f"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.733831 2388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:59.744314 2388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:59.746241 2388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:59.746381 2388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:59.750566 2388 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:45181
I20260927 07:06:59.750641 2455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:45181 every 8 connection(s)
I20260927 07:06:59.750965 2388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:59.751853 2456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:59.753379 2456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:59.759793 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2388
I20260927 07:06:59.759934 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:06:59.760128 21513 external_mini_cluster.cc:1428] Setting key 0d84afb2761155958e8a128f2794a5da
I20260927 07:06:59.763646 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.754123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.21.2.126:45181)
0927 07:06:59.754313 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:59.754321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:59.754473 (+ 152us) server_negotiation.cc:407] Connection header received
0927 07:06:59.755031 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:59.755038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:59.755055 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:59.755150 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:59.755767 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.756587 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.757496 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.757805 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.757971 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.760279 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:59.760287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:59.760288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:59.760296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:59.762036 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.762501 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.762503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.762504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.762578 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.762822 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.762823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.762823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.763018 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:59.763077 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:59.763295 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:06:59.763486 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260927 07:06:59.763774 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.754106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45181 (local address 127.0.0.1:34644)
0927 07:06:59.754304 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.754346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:59.754869 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:59.755158 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:59.755160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:59.755519 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:59.755669 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.755674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.756716 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.756717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.757394 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.757395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.757936 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.757937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.758004 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.758576 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:59.758584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:59.760164 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:59.762121 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.762129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.762131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.762414 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.762646 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.762646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.762647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.762757 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.763087 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:59.763094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:59.763172 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 07:06:59.763280 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:06:59.763578 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":82,"spinlock_wait_cycles":7552,"thread_start_us":100,"threads_started":1}
I20260927 07:06:59.764542 2456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:06:59.764905 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.764766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55076)
0927 07:06:59.764819 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.764870 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:06:59.765059 2406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
I20260927 07:06:59.765547 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.765426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55092)
0927 07:06:59.765479 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.765514 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 07:06:59.765707 2456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:33609: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
I20260927 07:06:59.784366 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:33609
--webserver_interface=127.21.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:06:59.814970 2456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } attempt: 1
I20260927 07:06:59.815565 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.815405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55100)
0927 07:06:59.815465 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.815515 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 07:06:59.815943 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.815845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55106)
0927 07:06:59.815893 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.815921 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:06:59.816069 2456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:33609: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
W20260927 07:06:59.867010 2463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:59.867226 2463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:59.867249 2463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:59.868449 2463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:59.868495 2463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:59.868510 2463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:59.868522 2463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:59.868564 2463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:59.870100 2463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:33609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:06:59.870432 2463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:59.870679 2463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:06:59.875667 2463 server_base.cc:1034] running on GCE node
W20260927 07:06:59.875635 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.875660 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.875633 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:59.876132 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:59.876451 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:59.877647 2463 hybrid_clock.cc:648] HybridClock initialized: now 1790492819877628 us; error 30 us; skew 500 ppm
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:59.880245 2463 init.cc:377] Logged in from keytab as kudu/127.21.2.125@KRBTEST.COM (short username kudu)
I20260927 07:06:59.880803 2463 webserver.cc:460] Webserver started at http://127.21.2.125:34511/ using document root <none> and password file <none>
I20260927 07:06:59.881081 2463 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:59.881162 2463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:59.881332 2463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:59.882524 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data/instance:
uuid: "b2cf69e4157d40efab6850b912fa269d"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "be1909d58b52b5f278b99ba77c1f21ea"
server_key_iv: "c6bc2c32db2046c86c47c6e37dab3a65"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.882951 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal/instance:
uuid: "b2cf69e4157d40efab6850b912fa269d"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "be1909d58b52b5f278b99ba77c1f21ea"
server_key_iv: "c6bc2c32db2046c86c47c6e37dab3a65"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.884234 2456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } attempt: 2
I20260927 07:06:59.884672 2463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 07:06:59.884697 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.884484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55114)
0927 07:06:59.884568 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.884646 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 07:06:59.885121 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.884997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55122)
0927 07:06:59.885055 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.885085 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 07:06:59.885268 2456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:33609: Network error: Client connection negotiation failed: client connection to 127.21.2.125:33609: connect: Connection refused (error 111)
I20260927 07:06:59.885572 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:59.885797 2463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:06:59.885924 2463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "b2cf69e4157d40efab6850b912fa269d"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-ltqz"
server_key: "be1909d58b52b5f278b99ba77c1f21ea"
server_key_iv: "c6bc2c32db2046c86c47c6e37dab3a65"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.886010 2463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:59.902077 2463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:59.904085 2463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:59.904266 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:59.908514 2463 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.125:33609
I20260927 07:06:59.908567 2530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.125:33609 every 8 connection(s)
I20260927 07:06:59.908974 2463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
I20260927 07:06:59.909265 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2463
I20260927 07:06:59.909381 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-1/wal/instance
I20260927 07:06:59.909621 21513 external_mini_cluster.cc:1428] Setting key 943323ffa1789fd85293b18d56350bc0
I20260927 07:06:59.910079 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:59.912143 2531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.124@KRBTEST.COM" created.
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:06:59.924360 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.912819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45181 (local address 127.0.0.1:34660)
0927 07:06:59.913004 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.913045 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:59.913555 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:59.913818 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:59.913821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:59.914063 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:59.914254 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.914258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.915279 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.915280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.916111 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.916113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.916637 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.916637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.916738 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.917378 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:59.917386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:59.921236 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:59.922903 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.922911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.922914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.923145 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.923432 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.923450 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.923450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.923562 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.923866 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:59.923870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:59.923996 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 07:06:59.924074 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:06:59.924198 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260927 07:06:59.924379 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.912893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.21.2.126:45181)
0927 07:06:59.912970 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:59.912973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:59.913158 (+ 185us) server_negotiation.cc:407] Connection header received
0927 07:06:59.913655 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:59.913657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:59.913673 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:59.913752 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:59.914349 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.915189 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.916201 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.916546 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.916627 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.921339 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:59.921360 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:59.921360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:59.921368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:59.922799 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.923224 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.923225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.923226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.923289 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.923639 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.923640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.923641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.923795 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:59.923869 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:59.924135 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 07:06:59.924235 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:06:59.925051 2531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:06:59.935138 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.925329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55134)
0927 07:06:59.925396 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.925408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:59.925456 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:59.925739 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:59.925740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:59.925850 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:59.925901 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.925902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.927243 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.927243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.927788 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.927789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.928422 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.928422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.928476 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.928878 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:59.928882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:59.932165 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:59.934156 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.934157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.934157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.934313 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.934573 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.934574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.934574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.934626 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.934902 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:59.934902 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:59.934939 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:06:59.934985 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 07:06:59.935069 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 07:06:59.935312 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.925409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.21.2.125:33609)
0927 07:06:59.925569 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:59.925574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:59.925600 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:59.925625 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:59.925627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:59.925643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:59.925731 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:59.925977 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.927154 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.927870 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.928340 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.928427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.932266 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:59.932276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:59.932277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:59.932285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:59.934073 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.934404 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.934430 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.934431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.934509 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.934687 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.934688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.934688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.934845 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:59.934913 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:59.935017 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 07:06:59.935132 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260927 07:06:59.936072 2531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:06:59.936487 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.936369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38012)
0927 07:06:59.936417 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.936461 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:06:59.936691 2479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
I20260927 07:06:59.937114 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.937054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38020)
0927 07:06:59.937076 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.937100 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
W20260927 07:06:59.937250 2531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:37503: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
I20260927 07:06:59.937889 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:37503
--webserver_interface=127.21.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:06:59.976464 2456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } attempt: 3
Sep 27 07:06:59 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:06:59.984292 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.976843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55146)
0927 07:06:59.976910 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.976922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:59.976999 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:59.977173 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:59.977174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:59.977341 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:59.977403 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.977405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.978091 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.978093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.978605 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:59.978607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.978996 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.978997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:59.979085 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.979632 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:59.979637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:59.981070 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:59.983069 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.983070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.983070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.983289 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.983607 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:59.983608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:59.983609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:59.983672 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:59.983947 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:59.983948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:59.984017 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 07:06:59.984089 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:06:59.984191 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 07:06:59.984406 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.976927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.21.2.125:33609)
0927 07:06:59.977002 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:59.977004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:59.977015 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:06:59.977080 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:59.977080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:59.977100 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:59.977166 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:59.977493 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.977993 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.978693 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.978903 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.978993 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.981170 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:59.981172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:59.981172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:59.981181 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:59.982956 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.983434 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.983436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.983436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.983525 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.983732 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.983733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.983734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.983886 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:59.983953 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:59.984138 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 07:06:59.984248 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:06:59.985145 2456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:06:59.985502 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.985400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38024)
0927 07:06:59.985446 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.985477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 07:06:59.985808 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.985758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38038)
0927 07:06:59.985779 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.985797 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 07:06:59.985905 2456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:37503: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
I20260927 07:06:59.986464 2531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } attempt: 1
I20260927 07:06:59.986790 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.986682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38050)
0927 07:06:59.986736 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.986763 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:06:59.987100 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.987008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38054)
0927 07:06:59.987058 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:59.987080 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 07:06:59.987186 2531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:37503: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
I20260927 07:07:00.017052 2456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } attempt: 1
I20260927 07:07:00.017536 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.017386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38060)
0927 07:07:00.017468 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.017500 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 07:07:00.017933 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.017849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38076)
0927 07:07:00.017893 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.017917 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:07:00.018025 2456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:37503: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
W20260927 07:07:00.021697 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:00.021888 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:00.021939 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:00.023094 2538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:00.023133 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:00.023149 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:00.023161 2538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:00.023212 2538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:00.024736 2538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45181,127.21.2.125:33609,127.21.2.124:37503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:37503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:00.025002 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:00.025247 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:00.030042 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:00.030042 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:00.030153 2538 server_base.cc:1034] running on GCE node
W20260927 07:07:00.030244 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:00.030550 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:00.030793 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:00.031929 2538 hybrid_clock.cc:648] HybridClock initialized: now 1790492820031921 us; error 36 us; skew 500 ppm
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:00.034396 2538 init.cc:377] Logged in from keytab as kudu/127.21.2.124@KRBTEST.COM (short username kudu)
I20260927 07:07:00.034813 2538 webserver.cc:460] Webserver started at http://127.21.2.124:41179/ using document root <none> and password file <none>
I20260927 07:07:00.035056 2538 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:00.035102 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:00.035254 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:00.036458 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data/instance:
uuid: "4134dfedd1fc42049c41f31d0f7fb11c"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-ltqz"
server_key: "6bdc4e1f15a7f841c47c8a478a687641"
server_key_iv: "b07e3b0fb953d9b0be6838b544531b83"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.036820 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal/instance:
uuid: "4134dfedd1fc42049c41f31d0f7fb11c"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-ltqz"
server_key: "6bdc4e1f15a7f841c47c8a478a687641"
server_key_iv: "b07e3b0fb953d9b0be6838b544531b83"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.038205 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:07:00.038939 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.039114 2538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:07:00.039212 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "4134dfedd1fc42049c41f31d0f7fb11c"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-ltqz"
server_key: "6bdc4e1f15a7f841c47c8a478a687641"
server_key_iv: "b07e3b0fb953d9b0be6838b544531b83"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.039309 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:00.055331 2531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } attempt: 2
I20260927 07:07:00.055884 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.055752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38082)
0927 07:07:00.055815 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.055856 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:07:00.056262 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.056162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38096)
0927 07:07:00.056210 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.056240 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:07:00.056355 2531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:37503: Network error: Client connection negotiation failed: client connection to 127.21.2.124:37503: connect: Connection refused (error 111)
I20260927 07:07:00.057700 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:00.059957 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:00.060112 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:00.064268 2538 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.124:37503
I20260927 07:07:00.064312 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.124:37503 every 8 connection(s)
I20260927 07:07:00.064697 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
I20260927 07:07:00.065645 2606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:00.067423 2606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:07:00.072757 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2538
I20260927 07:07:00.072841 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492641704908-21513-0/minicluster-data/master-2/wal/instance
I20260927 07:07:00.073037 21513 external_mini_cluster.cc:1428] Setting key 41f664353f8dd26bee56a06da0425c6b
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:00.078385 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.068204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.21.2.126:45181)
0927 07:07:00.068274 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.068277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.068500 (+ 223us) server_negotiation.cc:407] Connection header received
0927 07:07:00.069032 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.069033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.069043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.069087 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.069757 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.070214 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.071457 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.071644 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.071700 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.074640 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.074643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.074643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.074652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.076929 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.077376 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.077377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.077378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.077429 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.077708 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.077709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.077709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.077830 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.077910 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.078198 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.078294 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:07:00.078567 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.068175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:45181 (local address 127.0.0.1:34664)
0927 07:07:00.068374 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.068415 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.068925 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.069189 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.069193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.069473 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.069680 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.069684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.070328 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.070329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.071303 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.071306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.071725 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.071726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.071845 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.072480 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.072489 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.074483 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.077011 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.077018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.077019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.077299 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.077498 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.077499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.077500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.077645 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.077901 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.077908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.078050 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.078157 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.078317 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:07:00.079219 2606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:07:00.082686 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.073445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.21.2.124:37503)
0927 07:07:00.073577 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.073582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.073648 (+ 66us) server_negotiation.cc:407] Connection header received
0927 07:07:00.073742 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.073743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.073759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.073819 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.074659 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.075770 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.076710 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.077184 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.077237 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.079802 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.079808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.079808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.079812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.081468 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.081839 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.081842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.081842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.081893 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.082090 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.082091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.082091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.082271 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.082399 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.082497 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.082572 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":86,"threads_started":1}
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:07:00.087141 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.079551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:33609 (local address 127.0.0.1:55158)
0927 07:07:00.079602 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.079612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.079654 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.079848 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.079849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.079984 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.080033 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.080035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.080802 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.080803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.081594 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.081596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.082050 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.082051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.082139 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.082528 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.082533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.083940 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.086127 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.086128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.086145 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.086368 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.086583 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.086584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.086584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.086635 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.086916 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.086916 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.086949 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.086998 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.087075 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:07:00.087236 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.079642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.21.2.125:33609)
0927 07:07:00.079689 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.079691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.079699 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:00.079726 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.079727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.079737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.079805 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.080099 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.080725 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.081678 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.081976 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.082045 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.084022 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.084024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.084025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.084033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.086046 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.086440 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.086442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.086442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.086510 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.086697 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.086698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.086698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.086846 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.086924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.087029 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.087118 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:07:00.087867 2606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:00.091635 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.083586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.21.2.126:45181)
0927 07:07:00.083625 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.083627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.083635 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:00.083659 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.083660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.083670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.083738 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.084273 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.084710 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.085693 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.085873 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.085927 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.088709 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.088710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.088711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.088717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.090397 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.090767 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.090768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.090769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.090831 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.091038 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.091039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.091039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.091155 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.091370 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.091438 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.091522 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:07:00.093175 2456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } attempt: 2
I20260927 07:07:00.094973 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.088130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38112)
0927 07:07:00.088179 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.088191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.088229 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.088355 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.088355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.088468 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.088498 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.088499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.089286 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.089287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.090040 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.090042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.090443 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.090444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.090508 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.090910 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.090914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.092239 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.093915 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.093916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.093916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.094132 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.094337 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.094338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.094339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.094392 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.094640 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.094640 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.094678 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.094745 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.094841 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 07:07:00.094985 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.088203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.21.2.124:37503)
0927 07:07:00.088242 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.088244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.088254 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:07:00.088276 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.088276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.088287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.088354 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.088578 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.089205 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.090123 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.090370 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.090426 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.092320 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.092320 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.092321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.092326 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.093840 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.094212 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.094213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.094213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.094265 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.094464 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.094465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.094465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.094573 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.094649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.094771 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.094890 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":21}
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:07:00.097364 2606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: Bootstrap starting.
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:07:00.098026 2606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:00.098385 2606 log.cc:824] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:00.099577 2606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: No bootstrap required, opened a new log
I20260927 07:07:00.101238 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.092306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.21.2.125:33609)
0927 07:07:00.092345 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.092347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.092397 (+ 50us) server_negotiation.cc:407] Connection header received
0927 07:07:00.092469 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.092470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.092481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.092535 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.093358 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.094093 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.095031 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.095313 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.095351 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.098262 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.098263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.098264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.098271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.100022 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100385 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.100387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.100387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.100450 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100658 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.100659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.100659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.100811 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.101007 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.101046 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.101141 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:07:00.101323 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.093455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38122)
0927 07:07:00.093502 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.093513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.093550 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.093775 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.093775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.093955 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.094000 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.094002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.094938 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.094938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.095714 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.095716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.096205 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.096206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.096264 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.096720 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.096725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.098172 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.100179 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.100181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.100389 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.100630 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.100631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.100697 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.101032 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.101033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.101078 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.101128 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.101216 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 07:07:00.101776 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.093493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.21.2.124:37503)
0927 07:07:00.093652 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.093654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.093660 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:07:00.093678 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.093679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.093695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.093770 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.094097 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.094842 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.095817 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.096132 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.096203 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.098274 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.098276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.098276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.098284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.100038 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100469 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.100470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.100470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.100548 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.100773 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.100774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.100774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.100939 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.101494 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.101551 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.101648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260927 07:07:00.101929 2606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.102150 2606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:00.102254 2606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4134dfedd1fc42049c41f31d0f7fb11c, State: Initialized, Role: FOLLOWER
I20260927 07:07:00.102404 2606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.102610 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:07:00.102717 2613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.102768 2606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:00.102814 2613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.103253 2456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: Bootstrap starting.
I20260927 07:07:00.104102 2456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:00.104427 2456 log.cc:824] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: Log is configured to *not* fsync() on all Append() calls
W20260927 07:07:00.104854 2628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:07:00.104967 2628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:07:00.105285 2456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: No bootstrap required, opened a new log
I20260927 07:07:00.107488 2456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.107630 2456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:00.107666 2456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1c390f9db94fdc8e10c845f5e5c182, State: Initialized, Role: FOLLOWER
I20260927 07:07:00.107787 2456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.108093 2456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:00.105316 2538 master_runner.cc:186] Error getting master registration for 127.21.2.126:45181: OK, instance_id { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" instance_seqno: 1790492819749351 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:07:00.108678 2538 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:07:00.110219 2645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:07:00.110536 2645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:07:00.112665 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.104829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.21.2.126:45181)
0927 07:07:00.104881 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.104883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.104891 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:00.104910 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.104911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.104921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.104981 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.105726 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.106442 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.107514 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.107793 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.107838 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.109488 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.109489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.109490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.109497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.111265 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.111735 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.111736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.111737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.111792 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.112007 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.112008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.112008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.112135 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.112217 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.112462 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.112555 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:07:00.110915 2388 master_runner.cc:186] Error getting master registration for 127.21.2.125:33609: OK, instance_id { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" instance_seqno: 1790492819907260 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:07:00.113976 2388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:07:00.114110 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.114212 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.118702 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.104934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.21.2.125:33609)
0927 07:07:00.108773 (+ 3839us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.108776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.110905 (+ 2129us) server_negotiation.cc:407] Connection header received
0927 07:07:00.111014 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.111015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.111029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.111060 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.111720 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.112414 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.113531 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.113823 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.113867 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.115686 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.115687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.115687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.115696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.117311 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.117778 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.117780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.117780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.117837 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.118048 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.118050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.118050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.118182 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.118288 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.118502 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.118601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3809}
I20260927 07:07:00.118702 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.105135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.21.2.124:37503)
0927 07:07:00.105168 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.105170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.110677 (+ 5507us) server_negotiation.cc:407] Connection header received
0927 07:07:00.110803 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.110804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.110816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.110872 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.111619 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.112384 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.113531 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.113848 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.113883 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.115684 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.115685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.115685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.115693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.117469 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.117873 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.117874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.117875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.117928 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.118138 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.118139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.118139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.118257 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.118345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.118520 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.118602 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:07:00.147500 2531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } attempt: 3
Sep 27 07:07:00 dist-test-slave-ltqz krb5kdc[2375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:07:00.154968 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.147875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.21.2.124:37503)
0927 07:07:00.147926 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.147927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.147961 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:07:00.148057 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.148058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.148069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.148147 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.148468 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.148933 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.149612 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.149806 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.149884 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.151861 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.151863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.151863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.151871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.153580 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.154141 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.154142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.154142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.154207 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.154414 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.154415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.154416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.154537 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.154611 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.154779 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.154856 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 07:07:00.155015 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.147844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:37503 (local address 127.0.0.1:38144)
0927 07:07:00.147901 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.147914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.147978 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.148153 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.148154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.148325 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.148372 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.148374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.149016 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.149017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.149526 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.149528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.149880 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.149881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.149942 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.150426 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.150431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.151768 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.153677 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.153677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.153678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.153902 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.154279 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.154280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.154280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.154342 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.154615 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.154616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.154662 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.154742 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.154919 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:07:00.156553 2531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: Bootstrap starting.
I20260927 07:07:00.157210 2531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:00.157524 2531 log.cc:824] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:00.158288 2531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: No bootstrap required, opened a new log
I20260927 07:07:00.159802 2531 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.159940 2531 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:00.160001 2531 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cf69e4157d40efab6850b912fa269d, State: Initialized, Role: FOLLOWER
I20260927 07:07:00.160166 2531 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.160470 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.160539 2531 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:00.160605 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:07:00.162433 2659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:07:00.162540 2659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:07:00.165131 2463 master_runner.cc:263] No leader master found from master b2cf69e4157d40efab6850b912fa269d
I20260927 07:07:00.165179 2463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:07:00.247210 2409 tablet.cc:2369] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:00.310853 2559 tablet.cc:2369] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:00.405074 2484 tablet.cc:2369] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:00.407562 2613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:00.407693 2613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.408078 2613 leader_election.cc:288] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d1c390f9db94fdc8e10c845f5e5c182 (127.21.2.126:45181), b2cf69e4157d40efab6850b912fa269d (127.21.2.125:33609)
I20260927 07:07:00.408483 2505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cf69e4157d40efab6850b912fa269d" is_pre_election: true
I20260927 07:07:00.408619 2505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4134dfedd1fc42049c41f31d0f7fb11c in term 0.
I20260927 07:07:00.408432 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" is_pre_election: true
I20260927 07:07:00.408771 2430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4134dfedd1fc42049c41f31d0f7fb11c in term 0.
I20260927 07:07:00.408849 2556 leader_election.cc:302] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4134dfedd1fc42049c41f31d0f7fb11c, b2cf69e4157d40efab6850b912fa269d; no voters:
I20260927 07:07:00.409018 2613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:00.409094 2613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:00.409129 2613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:00.409979 2613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.410133 2613 leader_election.cc:288] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [CANDIDATE]: Term 1 election: Requested vote from peers 8d1c390f9db94fdc8e10c845f5e5c182 (127.21.2.126:45181), b2cf69e4157d40efab6850b912fa269d (127.21.2.125:33609)
I20260927 07:07:00.410277 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d1c390f9db94fdc8e10c845f5e5c182"
I20260927 07:07:00.410351 2505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2cf69e4157d40efab6850b912fa269d"
I20260927 07:07:00.410393 2430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:00.410466 2505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:00.411204 2430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4134dfedd1fc42049c41f31d0f7fb11c in term 1.
I20260927 07:07:00.411430 2505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4134dfedd1fc42049c41f31d0f7fb11c in term 1.
I20260927 07:07:00.411444 2554 leader_election.cc:302] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4134dfedd1fc42049c41f31d0f7fb11c, 8d1c390f9db94fdc8e10c845f5e5c182; no voters:
I20260927 07:07:00.411535 2613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:00.411741 2613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [term 1 LEADER]: Becoming Leader. State: Replica: 4134dfedd1fc42049c41f31d0f7fb11c, State: Running, Role: LEADER
I20260927 07:07:00.411850 2613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } }
I20260927 07:07:00.412336 2661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4134dfedd1fc42049c41f31d0f7fb11c. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.412484 2661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:00.412987 2663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:00.413199 2663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:00.414649 2430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [term 1 FOLLOWER]: Refusing update from remote peer 4134dfedd1fc42049c41f31d0f7fb11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:00.414969 2505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [term 1 FOLLOWER]: Refusing update from remote peer 4134dfedd1fc42049c41f31d0f7fb11c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:00.414968 2661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:00.415220 2661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:00.416325 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4134dfedd1fc42049c41f31d0f7fb11c. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.416352 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4134dfedd1fc42049c41f31d0f7fb11c. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.416436 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.416436 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.417269 2613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.417351 2613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:00.417392 2613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.417429 2613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:00.417526 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.417609 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.417712 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333858592415834112 to 7333858592407810048. Current Snapshot: MvccSnapshot[applied={T|T < 7333858592415834112}]
I20260927 07:07:00.417783 2664 mvcc.cc:205] Tried to move back new op lower bound from 7333858592415834112 to 7333858592407810048. Current Snapshot: MvccSnapshot[applied={T|T < 7333858592415834112}]
I20260927 07:07:00.417879 2666 mvcc.cc:205] Tried to move back new op lower bound from 7333858592415834112 to 7333858592407810048. Current Snapshot: MvccSnapshot[applied={T|T < 7333858592415834112}]
I20260927 07:07:00.418124 2663 catalog_manager.cc:1347] Generated new cluster ID: 1f532ebf32fb41e9b0965ec71d5f6d2c
I20260927 07:07:00.418097 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d1c390f9db94fdc8e10c845f5e5c182" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45181 } } peers { permanent_uuid: "b2cf69e4157d40efab6850b912fa269d" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 33609 } } peers { permanent_uuid: "4134dfedd1fc42049c41f31d0f7fb11c" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 37503 } } }
I20260927 07:07:00.418177 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:00.418181 2663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:00.431063 2663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:00.431762 2663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:00.435621 2663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4134dfedd1fc42049c41f31d0f7fb11c: Generated new TSK 0
I20260927 07:07:00.435868 2663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:07:01.111050 2645 catalog_manager.cc:1259] Loaded cluster ID: 1f532ebf32fb41e9b0965ec71d5f6d2c
I20260927 07:07:01.111125 2645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: loading cluster ID for follower catalog manager: success
I20260927 07:07:01.112043 2645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: acquiring CA information for follower catalog manager: success
I20260927 07:07:01.112349 2645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d1c390f9db94fdc8e10c845f5e5c182: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:07:01.162979 2659 catalog_manager.cc:1259] Loaded cluster ID: 1f532ebf32fb41e9b0965ec71d5f6d2c
I20260927 07:07:01.163055 2659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: loading cluster ID for follower catalog manager: success
I20260927 07:07:01.163895 2659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: acquiring CA information for follower catalog manager: success
I20260927 07:07:01.164214 2659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2cf69e4157d40efab6850b912fa269d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:07:01.499954 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2388
I20260927 07:07:01.505591 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2463
I20260927 07:07:01.510691 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2538
2026-09-27T07:07:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3046 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:01 dist-test-slave-ltqz krb5kdc[2678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:01 dist-test-slave-ltqz krb5kdc[2678](info): set up 2 sockets
Sep 27 07:07:01 dist-test-slave-ltqz krb5kdc[2678](info): commencing operation
krb5kdc: starting...
W20260927 07:07:02.551613 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:07:02.669842 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40385
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:02.749495 2691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:02.749701 2691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:02.749724 2691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:02.750813 2691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:02.750849 2691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:02.750864 2691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:02.750875 2691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:02.750887 2691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:02.752354 2691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:02.752550 2691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:02.752715 2691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:02.757403 2696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:02.757416 2699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.757715 2691 server_base.cc:1034] running on GCE node
W20260927 07:07:02.757422 2697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.758056 2691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:02.758363 2691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:02.759538 2691 hybrid_clock.cc:648] HybridClock initialized: now 1790492822759521 us; error 42 us; skew 500 ppm
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:02.766430 2691 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:02.766888 2691 webserver.cc:460] Webserver started at http://127.21.2.126:46357/ using document root <none> and password file <none>
I20260927 07:07:02.767118 2691 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:02.767184 2691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:02.767343 2691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:02.768466 2691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "8b0c9120fe4a45fb95956a9d29711451"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-ltqz"
server_key: "ec8034bd4ff0939dbdbdb6259d0cd9e6"
server_key_iv: "57b9b653f1e168e200872b4460d5e242"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.768834 2691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "8b0c9120fe4a45fb95956a9d29711451"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-ltqz"
server_key: "ec8034bd4ff0939dbdbdb6259d0cd9e6"
server_key_iv: "57b9b653f1e168e200872b4460d5e242"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.770157 2691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:07:02.770903 2706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:02.771085 2691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:07:02.771184 2691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "8b0c9120fe4a45fb95956a9d29711451"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-ltqz"
server_key: "ec8034bd4ff0939dbdbdb6259d0cd9e6"
server_key_iv: "57b9b653f1e168e200872b4460d5e242"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.771271 2691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:02.791769 2691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:02.793619 2691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:02.793787 2691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:02.797786 2691 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40385
I20260927 07:07:02.797822 2758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40385 every 8 connection(s)
I20260927 07:07:02.798184 2691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:02.799039 2759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:02.801323 2759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451: Bootstrap starting.
I20260927 07:07:02.801868 2759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:02.802094 2759 log.cc:824] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:02.802713 2759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451: No bootstrap required, opened a new log
I20260927 07:07:02.804229 2759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } }
I20260927 07:07:02.804363 2759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:02.804385 2759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0c9120fe4a45fb95956a9d29711451, State: Initialized, Role: FOLLOWER
I20260927 07:07:02.804535 2759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } }
I20260927 07:07:02.804626 2759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:02.804652 2759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:02.804725 2759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:02.804694 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2691
I20260927 07:07:02.804811 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:07:02.805022 21513 external_mini_cluster.cc:1428] Setting key c6aa1e9765dab9b797979c0fb726f3cc
I20260927 07:07:02.805415 2759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } }
I20260927 07:07:02.805604 2759 leader_election.cc:302] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0c9120fe4a45fb95956a9d29711451; no voters:
I20260927 07:07:02.805778 2759 leader_election.cc:288] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:02.805832 2764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:02.805971 2764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0c9120fe4a45fb95956a9d29711451, State: Running, Role: LEADER
I20260927 07:07:02.806023 2759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:02.806095 2764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } }
I20260927 07:07:02.806437 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b0c9120fe4a45fb95956a9d29711451. Latest consensus state: current_term: 1 leader_uuid: "8b0c9120fe4a45fb95956a9d29711451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } } }
I20260927 07:07:02.806572 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:02.806599 2765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b0c9120fe4a45fb95956a9d29711451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c9120fe4a45fb95956a9d29711451" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40385 } } }
I20260927 07:07:02.806782 2765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:02.806799 2772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:02.807331 2772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:02.808950 2772 catalog_manager.cc:1347] Generated new cluster ID: ddcc2d6f4e1244b2a77c8fd0cc4dfd65
I20260927 07:07:02.809020 2772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:02.816804 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.805478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.21.2.126:40385)
0927 07:07:02.805703 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.805714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.805751 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:07:02.805884 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.805893 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.805958 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.806970 (+ 1012us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.807101 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.808581 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.809690 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.810314 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.810429 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.813249 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.813253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.813253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.813264 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.815174 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.815599 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.815602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.815603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.815728 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.815944 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.815945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.815945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.816215 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.816356 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.816483 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.816606 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260927 07:07:02.818166 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:07:02.822614 2772 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:02.823127 2772 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:02.828418 2772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b0c9120fe4a45fb95956a9d29711451: Generated new TSK 0
I20260927 07:07:02.828616 2772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:07:02.872185 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2691
2026-09-27T07:07:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1359 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2786](info): set up 2 sockets
Sep 27 07:07:02 dist-test-slave-ltqz krb5kdc[2786](info): commencing operation
krb5kdc: starting...
W20260927 07:07:03.899585 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:07:03 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:07:04.016834 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:04.096844 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.097047 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.097069 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.098250 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:04.098304 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.098320 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.098331 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:04.098385 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:04.099867 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:04.100131 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.100368 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.105239 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.105285 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.105444 2799 server_base.cc:1034] running on GCE node
W20260927 07:07:04.105481 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.105705 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.105994 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.107172 2799 hybrid_clock.cc:648] HybridClock initialized: now 1790492824107159 us; error 27 us; skew 500 ppm
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.109688 2799 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:04.110144 2799 webserver.cc:460] Webserver started at http://127.21.2.126:37885/ using document root <none> and password file <none>
I20260927 07:07:04.110329 2799 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.110370 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.110459 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:04.111543 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "acae0a2eb5624f868fc665d6cf5c9cd8"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "051196b81573a259c6565d36236b1d97"
server_key_iv: "58b5e40c645624896da46e25bc2ef110"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.111857 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "acae0a2eb5624f868fc665d6cf5c9cd8"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "051196b81573a259c6565d36236b1d97"
server_key_iv: "58b5e40c645624896da46e25bc2ef110"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.113116 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:04.113796 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.113977 2799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:07:04.114084 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "acae0a2eb5624f868fc665d6cf5c9cd8"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "051196b81573a259c6565d36236b1d97"
server_key_iv: "58b5e40c645624896da46e25bc2ef110"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.114159 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.128602 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.130511 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.130683 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:04.134822 2799 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:35625
I20260927 07:07:04.134858 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:35625 every 8 connection(s)
I20260927 07:07:04.135241 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:04.136272 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.138543 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap starting.
I20260927 07:07:04.139134 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.139444 2867 log.cc:824] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.140131 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: No bootstrap required, opened a new log
I20260927 07:07:04.141721 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:04.141956 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.141959 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2799
I20260927 07:07:04.142002 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.142050 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:07:04.142166 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:04.142252 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:04.142259 21513 external_mini_cluster.cc:1428] Setting key 2f3bbc923f598873ec7c771c094137bd
I20260927 07:07:04.142290 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:04.142345 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:04.143463 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:04.143584 2867 leader_election.cc:302] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acae0a2eb5624f868fc665d6cf5c9cd8; no voters:
I20260927 07:07:04.143782 2867 leader_election.cc:288] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:04.143854 2872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:04.144007 2872 raft_consensus.cc:686] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 LEADER]: Becoming Leader. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Running, Role: LEADER
I20260927 07:07:04.144066 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:04.144133 2872 consensus_queue.cc:237] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:04.144451 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acae0a2eb5624f868fc665d6cf5c9cd8. Latest consensus state: current_term: 1 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:04.144543 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:04.144464 2873 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:04.144663 2873 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:04.144814 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:04.145321 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:04.146960 2879 catalog_manager.cc:1347] Generated new cluster ID: 27d59a5127854684a180f995020ec739
I20260927 07:07:04.147013 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:04.154028 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.142772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.21.2.126:35625)
0927 07:07:04.142934 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.142945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.142977 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:07:04.143072 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.143077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.143117 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.143226 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.144082 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.145580 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.146497 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.147078 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.147181 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.150080 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.150085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.150085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.150099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.152257 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.152657 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.152661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.152661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.152792 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.152994 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.152995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.152996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.153357 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.153559 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.153644 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.153791 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260927 07:07:04.165743 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:04.166478 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:04.169203 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Generated new TSK 0
I20260927 07:07:04.169404 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:07:04.196622 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:04.280078 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.280292 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.280316 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.281667 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.281713 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.281741 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:07:04.283208 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:04.283486 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.283727 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.288893 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.288887 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.288887 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.289119 2893 server_base.cc:1034] running on GCE node
I20260927 07:07:04.289423 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.289678 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.290848 2893 hybrid_clock.cc:648] HybridClock initialized: now 1790492824290832 us; error 46 us; skew 500 ppm
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.293292 2893 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:07:04.293708 2893 webserver.cc:460] Webserver started at http://127.21.2.65:39101/ using document root <none> and password file <none>
I20260927 07:07:04.293937 2893 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.294004 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.294157 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:04.295189 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "b1ab5e411f624456abf30e4851b27679"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "18e87efda9d01120234e57c8518fb58e"
server_key_iv: "938c86183e65d745990cb5959f60371e"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.295581 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "b1ab5e411f624456abf30e4851b27679"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "18e87efda9d01120234e57c8518fb58e"
server_key_iv: "938c86183e65d745990cb5959f60371e"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.296897 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:04.297598 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.297762 2893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:07:04.297868 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "b1ab5e411f624456abf30e4851b27679"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "18e87efda9d01120234e57c8518fb58e"
server_key_iv: "938c86183e65d745990cb5959f60371e"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.297951 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.322455 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.324426 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.324594 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:04.324837 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:04.325161 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:04.325222 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.325263 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:04.325291 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.331107 2893 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:42893
I20260927 07:07:04.331154 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:42893 every 8 connection(s)
I20260927 07:07:04.331578 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:04.340950 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.332335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:60113 (local address 127.21.2.126:35625)
0927 07:07:04.332433 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.332436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.332553 (+ 117us) server_negotiation.cc:407] Connection header received
0927 07:07:04.333029 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.333030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.333044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.333098 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.334043 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.334623 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.335784 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.335982 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.336042 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.338285 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.338286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.338287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.338294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.339728 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.340130 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.340131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.340131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.340181 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.340420 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.340420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.340420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.340523 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.340573 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.340812 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.340877 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:07:04.341192 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.332265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:60113)
0927 07:07:04.332427 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.332473 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.332945 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.333149 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.333152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.333724 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.333928 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.333932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.334751 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.334752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.335664 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.335668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.336060 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.336061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.336168 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.336740 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.336748 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.338197 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.339824 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.339832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.339833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.340062 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.340247 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.340247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.340247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.340363 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.340579 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.340583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.340701 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.340784 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.340906 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260927 07:07:04.341755 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 2893
I20260927 07:07:04.341796 3022 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:04.341892 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:04.341912 3022 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.342085 21513 external_mini_cluster.cc:1428] Setting key 32c254d783fa3b0a09647de27ba59fa4
I20260927 07:07:04.342109 3022 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:04.342643 2830 ts_manager.cc:194] Registered new tserver with Master: b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:04.343310 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:60113
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:07:04.366254 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:04.445482 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.445685 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.445709 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.446856 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.446902 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.446928 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:07:04.448370 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:04.448613 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.448884 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.453691 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.453697 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.453847 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.453963 3027 server_base.cc:1034] running on GCE node
I20260927 07:07:04.454156 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.454424 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.455567 3027 hybrid_clock.cc:648] HybridClock initialized: now 1790492824455542 us; error 40 us; skew 500 ppm
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.457983 3027 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:07:04.458437 3027 webserver.cc:460] Webserver started at http://127.21.2.66:34815/ using document root <none> and password file <none>
I20260927 07:07:04.458662 3027 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.458727 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.458884 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:04.460063 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "61d47d5d7132450a8bbe5195d394708a"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "010eb1a1ae2ee728fa25ed6a2ca81237"
server_key_iv: "6c9bbebaf91b6a5c54826fdff48cc36c"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.460427 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "61d47d5d7132450a8bbe5195d394708a"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "010eb1a1ae2ee728fa25ed6a2ca81237"
server_key_iv: "6c9bbebaf91b6a5c54826fdff48cc36c"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.461735 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:04.462540 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.462733 3027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:04.462836 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "61d47d5d7132450a8bbe5195d394708a"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "010eb1a1ae2ee728fa25ed6a2ca81237"
server_key_iv: "6c9bbebaf91b6a5c54826fdff48cc36c"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.462925 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.475911 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.477882 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.478039 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:04.478276 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:04.478585 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:04.478639 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.478683 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:04.478706 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.484488 3027 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:33407
I20260927 07:07:04.484570 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:33407 every 8 connection(s)
I20260927 07:07:04.484899 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:04.490947 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3027
I20260927 07:07:04.491041 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:04.491261 21513 external_mini_cluster.cc:1428] Setting key 2b249b8b8404cd02d00fc7400682381d
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:04.494757 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.485775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:40115 (local address 127.21.2.126:35625)
0927 07:07:04.485851 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.485853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.486060 (+ 207us) server_negotiation.cc:407] Connection header received
0927 07:07:04.486535 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.486536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.486545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.486603 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.487590 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.488069 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.489187 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.489378 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.489432 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.492012 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.492014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.492015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.492023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.493402 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.493813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.493814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.493815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.493875 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.494124 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.494125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.494125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.494240 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.494312 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.494586 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.494667 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:07:04.494964 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.485748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:40115)
0927 07:07:04.485928 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.485971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.486452 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.486655 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.486667 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.487231 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.487511 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.487516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.488185 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.488186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.489077 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.489081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.489470 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.489471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.489588 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.490190 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.490201 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.491918 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.493486 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.493492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.493495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.493713 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.493941 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.493942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.493942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.494054 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.494315 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.494321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.494457 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.494565 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.494718 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260927 07:07:04.495440 3156 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:04.495543 3156 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.495741 3156 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:04.496169 2830 ts_manager.cc:194] Registered new tserver with Master: 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407)
I20260927 07:07:04.496670 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:40115
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:07:04.518008 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:04.597612 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.597824 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.597847 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.598997 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.599042 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.599069 3161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:07:04.600566 3161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:04.600817 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.601058 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.605949 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.605959 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.605993 3161 server_base.cc:1034] running on GCE node
W20260927 07:07:04.605959 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.606393 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.606601 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.607733 3161 hybrid_clock.cc:648] HybridClock initialized: now 1790492824607722 us; error 30 us; skew 500 ppm
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.610304 3161 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:07:04.610775 3161 webserver.cc:460] Webserver started at http://127.21.2.67:40797/ using document root <none> and password file <none>
I20260927 07:07:04.611012 3161 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.611057 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.611210 3161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:04.612288 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "351e5955901d4337ae52de9ebe207d23"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "440a985df2e11b7139a2ff6c454867f5"
server_key_iv: "919a2d6d6efa6f6036197636b55cd303"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.612629 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "351e5955901d4337ae52de9ebe207d23"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "440a985df2e11b7139a2ff6c454867f5"
server_key_iv: "919a2d6d6efa6f6036197636b55cd303"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.613993 3161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:04.614796 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.615142 3161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:04.615209 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "351e5955901d4337ae52de9ebe207d23"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "440a985df2e11b7139a2ff6c454867f5"
server_key_iv: "919a2d6d6efa6f6036197636b55cd303"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.615288 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.623245 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.625460 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.625610 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:04.625829 3161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:04.626132 3161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:04.626183 3161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.626217 3161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:04.626245 3161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.632005 3161 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:39487
I20260927 07:07:04.632056 3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:39487 every 8 connection(s)
I20260927 07:07:04.632355 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:04.632916 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3161
I20260927 07:07:04.633023 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:04.633257 21513 external_mini_cluster.cc:1428] Setting key 6e20b277d8cb315b1388d5466f624ddf
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:04.641743 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.633276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48639 (local address 127.21.2.126:35625)
0927 07:07:04.633337 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.633339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.633462 (+ 123us) server_negotiation.cc:407] Connection header received
0927 07:07:04.634150 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.634151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.634161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.634256 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.635197 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.635769 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.636700 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.636890 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.636943 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.638902 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.638904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.638904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.638912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.640405 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.640805 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.640806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.640807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.640867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.641113 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.641114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.641114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.641232 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.641309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.641577 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.641654 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:07:04.641920 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.633028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:48639)
0927 07:07:04.633238 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.633287 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.634000 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.634319 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.634323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.634934 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.635116 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.635121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.635851 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.635852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.636624 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.636626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.636962 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.636963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.637060 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.637475 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.637482 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.638811 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.640504 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.640511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.640513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.640726 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.640946 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.640947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.640947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.641048 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.641302 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.641308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.641444 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.641542 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.641693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
I20260927 07:07:04.642335 3290 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:04.642445 3290 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.642617 3290 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:04.642989 2830 ts_manager.cc:194] Registered new tserver with Master: 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
I20260927 07:07:04.643410 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:48639
I20260927 07:07:04.645334 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:07:04.652309 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.646765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.21.2.126:35625)
0927 07:07:04.646835 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.646837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.646956 (+ 119us) server_negotiation.cc:407] Connection header received
0927 07:07:04.646994 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.646995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.647003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.647050 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.647645 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.648097 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.648790 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.648973 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.649023 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.650071 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.650072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.650072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.650078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.651462 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.651739 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.651739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.651739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.651792 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.651933 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.651933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.651933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.652049 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.652114 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.652195 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.652249 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:07:04.653698 2830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:04.655261 2830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:04.661270 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:33407 (local address 127.0.0.1:37794)
0927 07:07:04.657503 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.657543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.657608 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.657877 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.657879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.657880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.657960 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.657962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.659005 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.659006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.660054 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.660057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.660932 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.660933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.661053 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.661068 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.661143 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.661195 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":54,"threads_started":1}
I20260927 07:07:04.661293 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.21.2.66:33407)
0927 07:07:04.657707 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.657711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.657723 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:07:04.657765 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.657767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.657784 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.657878 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.658026 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.658909 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.660145 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.660852 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.660917 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.660970 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.661173 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.661211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260927 07:07:04.661990 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.0.0.1:48686)
0927 07:07:04.657684 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.657694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.657740 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.658227 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.658228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.658230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.658281 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.658283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.659621 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.659623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.660627 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.660630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.661751 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.661752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.661818 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.661821 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.661880 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.661920 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":60,"threads_started":1}
I20260927 07:07:04.662003 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.21.2.65:42893)
0927 07:07:04.657995 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.658061 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.658072 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:07:04.658112 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.658114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.658130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.658373 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.658446 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.659514 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.660710 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.661665 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.661749 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.661803 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.661893 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.661929 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260927 07:07:04.662067 3090 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (DEFAULT_TABLE table=test-table [id=9d85a6872bcb4b26814bba9124c9b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.662442 3090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.663184 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:39487 (local address 127.0.0.1:52742)
0927 07:07:04.657703 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.657714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.657750 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.658230 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.658231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.658233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.658283 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.658285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.660715 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.660716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.661855 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.661858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.663012 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.663013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.663060 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.663064 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.663099 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.663141 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":63,"threads_started":1}
I20260927 07:07:04.663404 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.657598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.21.2.67:39487)
0927 07:07:04.658017 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.658021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.658030 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:07:04.658064 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.658066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.658080 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.658155 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.658358 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.659514 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.661955 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.662919 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.662989 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.663036 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.663122 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.663171 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
I20260927 07:07:04.663961 3224 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (DEFAULT_TABLE table=test-table [id=9d85a6872bcb4b26814bba9124c9b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.664259 3224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.665711 3304 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Bootstrap starting.
I20260927 07:07:04.666208 3304 tablet_bootstrap.cc:654] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.666113 2956 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (DEFAULT_TABLE table=test-table [id=9d85a6872bcb4b26814bba9124c9b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.666422 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.666592 3304 log.cc:824] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.667347 3304 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: No bootstrap required, opened a new log
I20260927 07:07:04.667440 3304 ts_tablet_manager.cc:1397] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:04.667626 3306 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Bootstrap starting.
I20260927 07:07:04.668298 3306 tablet_bootstrap.cc:654] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.668905 3306 log.cc:824] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.669657 3308 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Bootstrap starting.
I20260927 07:07:04.669651 3304 raft_consensus.cc:348] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.669902 3306 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: No bootstrap required, opened a new log
I20260927 07:07:04.669926 3304 raft_consensus.cc:374] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.669963 3306 ts_tablet_manager.cc:1397] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:04.669966 3304 raft_consensus.cc:729] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d47d5d7132450a8bbe5195d394708a, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.670109 3304 consensus_queue.cc:260] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.670394 3308 tablet_bootstrap.cc:654] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.670397 3156 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
I20260927 07:07:04.670637 3304 ts_tablet_manager.cc:1428] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:04.670975 3308 log.cc:824] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.671751 3308 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: No bootstrap required, opened a new log
I20260927 07:07:04.671841 3308 ts_tablet_manager.cc:1397] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:04.672034 3306 raft_consensus.cc:348] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.672178 3306 raft_consensus.cc:374] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.672221 3306 raft_consensus.cc:729] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e5955901d4337ae52de9ebe207d23, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.672339 3306 consensus_queue.cc:260] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.672617 3306 ts_tablet_manager.cc:1428] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:07:04.672650 3290 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
I20260927 07:07:04.673779 3308 raft_consensus.cc:348] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.673903 3308 raft_consensus.cc:374] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.673923 3308 raft_consensus.cc:729] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ab5e411f624456abf30e4851b27679, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.674024 3308 consensus_queue.cc:260] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.674248 3308 ts_tablet_manager.cc:1428] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:07:04.674311 3022 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
W20260927 07:07:04.735677 3157 tablet.cc:2369] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:04.801913 3311 raft_consensus.cc:482] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:04.802198 3311 raft_consensus.cc:504] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.802680 3311 leader_election.cc:288] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:04.805444 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.802840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:33407 (local address 127.21.2.67:48459)
0927 07:07:04.802900 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.802913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.802960 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.803109 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.803110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.803111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.803190 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.803192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.803887 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.803888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.804620 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.804622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.805175 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.805176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.805260 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.805263 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.805331 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.805372 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:07:04.805444 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.802938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:48459 (local address 127.21.2.66:33407)
0927 07:07:04.802984 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.802986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.802998 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:07:04.803026 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.803027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.803038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.803094 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.803271 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.803798 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.804716 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.805089 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.805175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.805187 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.805345 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.805384 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:07:04.805540 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.802853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.21.2.67:36577)
0927 07:07:04.803026 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.803035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.803076 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.803245 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.803246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.803247 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.803336 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.803338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.803985 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.803986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.804706 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.804708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.805272 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.805273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.805354 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.805357 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.805422 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.805460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260927 07:07:04.805531 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.803036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:36577 (local address 127.21.2.65:42893)
0927 07:07:04.803116 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.803118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.803127 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:07:04.803152 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.803153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.803172 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.803240 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.803446 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.803900 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.804802 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.805174 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.805269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.805279 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.805428 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.805465 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:07:04.805799 3110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7ed4be1d5e48abae4eff6b255c2d2f" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d47d5d7132450a8bbe5195d394708a" is_pre_election: true
I20260927 07:07:04.805970 3110 raft_consensus.cc:2454] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 0.
I20260927 07:07:04.805980 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7ed4be1d5e48abae4eff6b255c2d2f" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ab5e411f624456abf30e4851b27679" is_pre_election: true
I20260927 07:07:04.806142 2976 raft_consensus.cc:2454] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 0.
I20260927 07:07:04.806197 3177 leader_election.cc:302] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a; no voters:
I20260927 07:07:04.806315 3311 raft_consensus.cc:2781] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:04.806389 3311 raft_consensus.cc:482] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:04.806416 3311 raft_consensus.cc:3037] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:04.807111 3311 raft_consensus.cc:504] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.807250 3311 leader_election.cc:288] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 election: Requested vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:04.807451 2976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7ed4be1d5e48abae4eff6b255c2d2f" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ab5e411f624456abf30e4851b27679"
I20260927 07:07:04.807518 3110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7ed4be1d5e48abae4eff6b255c2d2f" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d47d5d7132450a8bbe5195d394708a"
I20260927 07:07:04.807552 2976 raft_consensus.cc:3037] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:04.807614 3110 raft_consensus.cc:3037] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:04.808267 2976 raft_consensus.cc:2454] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 1.
I20260927 07:07:04.808403 3110 raft_consensus.cc:2454] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 1.
I20260927 07:07:04.808521 3180 leader_election.cc:302] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5955901d4337ae52de9ebe207d23, b1ab5e411f624456abf30e4851b27679; no voters:
I20260927 07:07:04.808604 3311 raft_consensus.cc:2781] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:04.808737 3311 raft_consensus.cc:686] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 1 LEADER]: Becoming Leader. State: Replica: 351e5955901d4337ae52de9ebe207d23, State: Running, Role: LEADER
I20260927 07:07:04.808846 3311 consensus_queue.cc:237] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:04.809662 2830 catalog_manager.cc:5495] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 351e5955901d4337ae52de9ebe207d23 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "351e5955901d4337ae52de9ebe207d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } health_report { overall_health: UNKNOWN } } }
W20260927 07:07:04.832252 3023 tablet.cc:2369] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:04.843118 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.840904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.21.2.67:39487)
0927 07:07:04.840966 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.840968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.840976 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:04.841026 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.841027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.841046 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.841098 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:04.841214 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.841698 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.842405 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.842585 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.842634 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.842761 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:04.842854 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:04.842913 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.842953 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.842971 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:07:04.845198 3110 raft_consensus.cc:1261] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Refusing update from remote peer 351e5955901d4337ae52de9ebe207d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:04.845257 2976 raft_consensus.cc:1261] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Refusing update from remote peer 351e5955901d4337ae52de9ebe207d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:04.845615 3314 consensus_queue.cc:1035] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:04.845858 3311 consensus_queue.cc:1035] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:04.848134 3317 mvcc.cc:205] Tried to move back new op lower bound from 7333858610562826240 to 7333858610417831936. Current Snapshot: MvccSnapshot[applied={T|T < 7333858610562826240}]
I20260927 07:07:04.848721 3318 mvcc.cc:205] Tried to move back new op lower bound from 7333858610562826240 to 7333858610417831936. Current Snapshot: MvccSnapshot[applied={T|T < 7333858610562826240}]
I20260927 07:07:04.850165 3321 mvcc.cc:205] Tried to move back new op lower bound from 7333858610562826240 to 7333858610417831936. Current Snapshot: MvccSnapshot[applied={T|T < 7333858610562826240}]
I20260927 07:07:04.850243 2830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:04.850342 2830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:04.851132 2830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Sending DeleteTablet for 3 replicas of tablet 9e7ed4be1d5e48abae4eff6b255c2d2f
I20260927 07:07:04.851475 3090 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:04 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:37794
I20260927 07:07:04.851495 3224 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:04 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:52742
I20260927 07:07:04.851536 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:04 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48686
I20260927 07:07:04.851711 3327 tablet_replica.cc:330] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: stopping tablet replica
I20260927 07:07:04.851799 3327 raft_consensus.cc:2229] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:04.851852 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2893
I20260927 07:07:04.851898 3327 raft_consensus.cc:2258] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:04.853545 3327 ts_tablet_manager.cc:1905] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:04.855209 3327 ts_tablet_manager.cc:1918] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:04.855314 3327 log.cc:1196] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/9e7ed4be1d5e48abae4eff6b255c2d2f
I20260927 07:07:04.855605 3327 ts_tablet_manager.cc:1939] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 61d47d5d7132450a8bbe5195d394708a: Deleting consensus metadata
I20260927 07:07:04.856014 2815 catalog_manager.cc:4841] TS 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (table test-table [id=9d85a6872bcb4b26814bba9124c9b370]) successfully deleted
W20260927 07:07:04.857941 2818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:42893 (error 108)
I20260927 07:07:04.858428 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.858298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.0.0.1:48694)
0927 07:07:04.858354 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.858390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:07:04.858460 3326 tablet_replica.cc:330] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: stopping tablet replica
I20260927 07:07:04.858563 3326 raft_consensus.cc:2229] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 1 LEADER]: Raft consensus shutting down.
W20260927 07:07:04.858646 2818 catalog_manager.cc:4584] TS b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f: Network error: Client connection negotiation failed: client connection to 127.21.2.65:42893: connect: Connection refused (error 111)
I20260927 07:07:04.858700 3326 raft_consensus.cc:2258] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:04.858881 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3027
I20260927 07:07:04.859202 3326 ts_tablet_manager.cc:1905] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:04.861088 3326 ts_tablet_manager.cc:1918] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:04.861163 3326 log.cc:1196] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/9e7ed4be1d5e48abae4eff6b255c2d2f
I20260927 07:07:04.861464 3326 ts_tablet_manager.cc:1939] T 9e7ed4be1d5e48abae4eff6b255c2d2f P 351e5955901d4337ae52de9ebe207d23: Deleting consensus metadata
I20260927 07:07:04.861879 2817 catalog_manager.cc:4841] TS 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (table test-table [id=9d85a6872bcb4b26814bba9124c9b370]) successfully deleted
I20260927 07:07:04.864883 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3161
I20260927 07:07:04.870035 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 2799
I20260927 07:07:04.874918 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--webserver_interface=127.21.2.126
--webserver_port=37885
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:04.956089 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.956302 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.956326 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.957423 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:04.957461 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.957475 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.957489 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:04.957500 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:04.958897 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:04.959079 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.959249 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.967867 3339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.967854 3336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.967849 3337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.968293 3330 server_base.cc:1034] running on GCE node
I20260927 07:07:04.968569 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.968888 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.970052 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790492824970037 us; error 44 us; skew 500 ppm
Sep 27 07:07:04 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.972703 3330 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:04.973150 3330 webserver.cc:460] Webserver started at http://127.21.2.126:37885/ using document root <none> and password file <none>
I20260927 07:07:04.973412 3330 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.973495 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.974857 3330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:04.975556 3346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.975726 3330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:07:04.975832 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "acae0a2eb5624f868fc665d6cf5c9cd8"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "051196b81573a259c6565d36236b1d97"
server_key_iv: "58b5e40c645624896da46e25bc2ef110"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.976258 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.993932 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.996349 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.996510 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.000484 3330 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:35625
I20260927 07:07:05.000520 3398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:35625 every 8 connection(s)
I20260927 07:07:05.000948 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:05.002941 3399 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:05.004010 3399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap starting.
I20260927 07:07:05.010423 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3330
I20260927 07:07:05.010726 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:42893
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=39101
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:05.012526 3399 log.cc:824] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.014434 3399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:05.014739 3399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap complete.
I20260927 07:07:05.017047 3399 raft_consensus.cc:348] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.017297 3399 raft_consensus.cc:729] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.017503 3399 consensus_queue.cc:260] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.017599 3399 raft_consensus.cc:388] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:05.017659 3399 raft_consensus.cc:482] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:05.017707 3399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:07:05.018710 3399 raft_consensus.cc:504] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.018838 3399 leader_election.cc:302] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acae0a2eb5624f868fc665d6cf5c9cd8; no voters:
I20260927 07:07:05.019039 3399 leader_election.cc:288] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:07:05.019078 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:07:05.019200 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 LEADER]: Becoming Leader. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Running, Role: LEADER
I20260927 07:07:05.019318 3399 sys_catalog.cc:564] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:05.019285 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.019606 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:05.019693 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:05.019591 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acae0a2eb5624f868fc665d6cf5c9cd8. Latest consensus state: current_term: 2 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:05.019865 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:05.020342 3407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:05.021124 3407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:05.021221 3407 catalog_manager.cc:1259] Loaded cluster ID: 27d59a5127854684a180f995020ec739
I20260927 07:07:05.021260 3407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:05.021602 3407 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 07:07:05.026650 3407 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 07:07:05.029520 3407 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:05.029711 3407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Loaded TSK: 0
I20260927 07:07:05.029917 3407 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:07:05.094614 3401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:05.094830 3401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:05.094856 3401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:05.096095 3401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:05.096143 3401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:05.096189 3401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:07:05.097678 3401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:42893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=39101
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:05.097945 3401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:05.098206 3401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:05.103202 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.103287 3401 server_base.cc:1034] running on GCE node
W20260927 07:07:05.103382 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.103214 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.103602 3401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:05.103853 3401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:05.104995 3401 hybrid_clock.cc:648] HybridClock initialized: now 1790492825104983 us; error 30 us; skew 500 ppm
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:05.107501 3401 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:07:05.107935 3401 webserver.cc:460] Webserver started at http://127.21.2.65:39101/ using document root <none> and password file <none>
I20260927 07:07:05.108181 3401 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:05.108242 3401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:05.109629 3401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.110373 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.110571 3401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.110669 3401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "b1ab5e411f624456abf30e4851b27679"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "18e87efda9d01120234e57c8518fb58e"
server_key_iv: "938c86183e65d745990cb5959f60371e"
server_key_version: "encryptionkey@0"
I20260927 07:07:05.111023 3401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:05.134228 3401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:05.136631 3401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:05.136821 3401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.137092 3401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:05.137610 3443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:07:05.139024 3401 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:07:05.139079 3401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:07:05.139115 3401 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:07:05.139851 3401 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:07:05.139897 3401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:07:05.139940 3443 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Bootstrap starting.
I20260927 07:07:05.147807 3401 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:42893
I20260927 07:07:05.147924 3550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:42893 every 8 connection(s)
I20260927 07:07:05.148366 3401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:05.151698 3443 log.cc:824] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.154132 3443 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:05.154428 3443 tablet_bootstrap.cc:492] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Bootstrap complete.
I20260927 07:07:05.155642 3443 ts_tablet_manager.cc:1397] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260927 07:07:05.156095 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3401
I20260927 07:07:05.156371 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:33407
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=34815
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:05.156961 3443 raft_consensus.cc:348] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.157323 3443 raft_consensus.cc:729] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ab5e411f624456abf30e4851b27679, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.157464 3443 consensus_queue.cc:260] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.157737 3443 ts_tablet_manager.cc:1428] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:07:05.160760 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.149193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:35033 (local address 127.21.2.126:35625)
0927 07:07:05.149452 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.149459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.149527 (+ 68us) server_negotiation.cc:407] Connection header received
0927 07:07:05.150464 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.150469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.150509 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.150615 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.151613 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.152535 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.153971 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.154413 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.154515 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.157182 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.157186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.157187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.157199 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.158991 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.159508 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.159510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.159511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.159600 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.159894 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.159894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.159895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.160098 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.160156 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.160516 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.160622 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":76,"threads_started":1}
I20260927 07:07:05.160897 3553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.149133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:35033)
0927 07:07:05.149378 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.149435 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.150267 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.150706 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.150712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.151170 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:05.151491 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.151500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.152656 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.152657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.153880 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.153882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.154517 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.154518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.154677 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.155221 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:05.155231 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:05.157063 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:05.159066 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.159072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.159074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.159422 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.159670 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.159671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.159671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.159822 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.160169 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:05.160175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:05.160339 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.160450 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.160637 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
I20260927 07:07:05.161631 3551 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:05.161747 3551 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:05.161968 3551 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:05.162701 3363 ts_manager.cc:194] Registered new tserver with Master: b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:05.163741 3363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:35033
I20260927 07:07:05.164949 3551 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
I20260927 07:07:05.170848 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.163270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.0.0.1:48702)
0927 07:07:05.167469 (+ 4199us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.167499 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.167560 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.167737 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.167739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.167741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.167792 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.167793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.168990 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.168990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.169764 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.169766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.170628 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.170629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.170690 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.170700 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.170750 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.170791 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4175,"thread_start_us":111,"threads_started":1}
I20260927 07:07:05.170877 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.163349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.21.2.65:42893)
0927 07:07:05.163537 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.163542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.167569 (+ 4027us) server_negotiation.cc:407] Connection header received
0927 07:07:05.167635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.167638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.167653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.167736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.167839 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.168915 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.169842 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.170553 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.170626 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.170679 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.170771 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.170806 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260927 07:07:05.171514 3485 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7ed4be1d5e48abae4eff6b255c2d2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:04 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48702
I20260927 07:07:05.172021 3560 tablet_replica.cc:330] stopping tablet replica
I20260927 07:07:05.172127 3560 raft_consensus.cc:2229] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:05.172194 3560 raft_consensus.cc:2258] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.172499 3560 ts_tablet_manager.cc:1905] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.174855 3560 ts_tablet_manager.cc:1918] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.174933 3560 log.cc:1196] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/9e7ed4be1d5e48abae4eff6b255c2d2f
I20260927 07:07:05.175156 3560 ts_tablet_manager.cc:1939] T 9e7ed4be1d5e48abae4eff6b255c2d2f P b1ab5e411f624456abf30e4851b27679: Deleting consensus metadata
I20260927 07:07:05.175762 3350 catalog_manager.cc:4841] TS b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893): tablet 9e7ed4be1d5e48abae4eff6b255c2d2f (table test-table [id=9d85a6872bcb4b26814bba9124c9b370]) successfully deleted
W20260927 07:07:05.243710 3556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:05.243947 3556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:05.243984 3556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:05.245126 3556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:05.245193 3556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:05.245249 3556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:07:05.246833 3556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=34815
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:05.247092 3556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:05.247334 3556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:05.252250 3565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.252229 3566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.252242 3568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.252487 3556 server_base.cc:1034] running on GCE node
I20260927 07:07:05.252689 3556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:05.252954 3556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:05.254102 3556 hybrid_clock.cc:648] HybridClock initialized: now 1790492825254080 us; error 37 us; skew 500 ppm
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:05.256695 3556 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:07:05.257126 3556 webserver.cc:460] Webserver started at http://127.21.2.66:34815/ using document root <none> and password file <none>
I20260927 07:07:05.257377 3556 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:05.257462 3556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:05.258791 3556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.259447 3575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.259617 3556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.259713 3556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "61d47d5d7132450a8bbe5195d394708a"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "010eb1a1ae2ee728fa25ed6a2ca81237"
server_key_iv: "6c9bbebaf91b6a5c54826fdff48cc36c"
server_key_version: "encryptionkey@0"
I20260927 07:07:05.260056 3556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:05.280093 3556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:05.282572 3556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:05.282747 3556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.282995 3556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:05.283310 3556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:05.283366 3556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.283429 3556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:05.283455 3556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.289186 3556 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:33407
I20260927 07:07:05.289242 3688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:33407 every 8 connection(s)
I20260927 07:07:05.289564 3556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:05.292382 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3556
I20260927 07:07:05.292652 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:39487
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=40797
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:05.299293 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.290251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:37113 (local address 127.21.2.126:35625)
0927 07:07:05.290342 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.290344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.290535 (+ 191us) server_negotiation.cc:407] Connection header received
0927 07:07:05.291095 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.291097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.291109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.291170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.292127 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.292794 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.293876 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.294078 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.294137 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.296476 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.296477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.296478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.296492 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.297882 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.298289 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.298290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.298291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.298353 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.298609 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.298610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.298611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.298748 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.298820 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.299102 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.299191 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:07:05.299484 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.290218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:37113)
0927 07:07:05.290411 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.290447 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.290986 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.291241 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.291245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.291835 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:05.292022 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.292026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.292887 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.292888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.293767 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.293770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.294166 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.294167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.294322 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.294834 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:05.294844 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:05.296370 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:05.297982 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.297989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.297991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.298209 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.298430 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.298430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.298431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.298533 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.298825 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:05.298831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:05.298966 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.299068 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.299227 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260927 07:07:05.300004 3689 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:05.300118 3689 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:05.300335 3689 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:05.300951 3363 ts_manager.cc:194] Registered new tserver with Master: 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407)
I20260927 07:07:05.301370 3363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:37113
W20260927 07:07:05.374574 3692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:05.374775 3692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:05.374799 3692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:05.375983 3692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:05.376025 3692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:05.376052 3692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:07:05.377539 3692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:39487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=40797
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:05.377774 3692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:05.378018 3692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:05.383082 3700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.383082 3698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.383159 3697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.383097 3692 server_base.cc:1034] running on GCE node
I20260927 07:07:05.383615 3692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:05.383889 3692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:05.385030 3692 hybrid_clock.cc:648] HybridClock initialized: now 1790492825385009 us; error 37 us; skew 500 ppm
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:05.387686 3692 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:07:05.388159 3692 webserver.cc:460] Webserver started at http://127.21.2.67:40797/ using document root <none> and password file <none>
I20260927 07:07:05.388399 3692 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:05.388454 3692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:05.389879 3692 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:05.390547 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.390741 3692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:05.390841 3692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "351e5955901d4337ae52de9ebe207d23"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "440a985df2e11b7139a2ff6c454867f5"
server_key_iv: "919a2d6d6efa6f6036197636b55cd303"
server_key_version: "encryptionkey@0"
I20260927 07:07:05.391202 3692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:05.412715 3692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:05.414772 3692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:05.414938 3692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.415177 3692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:05.415532 3692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:05.415596 3692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.415639 3692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:05.415668 3692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.421813 3692 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:39487
I20260927 07:07:05.421864 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:39487 every 8 connection(s)
I20260927 07:07:05.422194 3692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:05.428220 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3692
I20260927 07:07:05.431519 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.422960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:46445 (local address 127.21.2.126:35625)
0927 07:07:05.423026 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.423028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.423217 (+ 189us) server_negotiation.cc:407] Connection header received
0927 07:07:05.423717 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.423718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.423728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.423786 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.424717 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.425220 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.426178 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.426360 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.426410 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.428526 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.428527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.428528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.428534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.430115 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.430563 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.430565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.430565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.430628 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.430856 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.430857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.430858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.430986 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.431052 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.431328 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.431443 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:07:05.431720 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.422919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:46445)
0927 07:07:05.423080 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.423121 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.423625 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.423840 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.423843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.424418 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:05.424611 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.424615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.425317 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.425317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.426100 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.426101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.426436 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.426436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.426534 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.426999 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:05.427007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:05.428419 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:05.430194 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.430201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.430203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.430494 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.430696 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:05.430697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:05.430697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:05.430791 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:05.431052 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:05.431056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:05.431167 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.431288 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.431465 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260927 07:07:05.432215 3821 heartbeater.cc:347] Connected to a master server at 127.21.2.126:35625
I20260927 07:07:05.432334 3821 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:05.432533 3821 heartbeater.cc:511] Master 127.21.2.126:35625 requested a full tablet report, sending...
I20260927 07:07:05.432919 3363 ts_manager.cc:194] Registered new tserver with Master: 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
I20260927 07:07:05.433316 3363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:46445
I20260927 07:07:05.435563 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.428662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.21.2.126:35625)
0927 07:07:05.428760 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.428762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.428807 (+ 45us) server_negotiation.cc:407] Connection header received
0927 07:07:05.428904 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.428905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.428917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.428986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.429755 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.430289 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.431263 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.431586 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.431650 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.433269 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.433271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.433271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.433277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.434651 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.434953 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.434953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.434954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.435001 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.435147 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.435147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.435147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.435229 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.435338 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.435431 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.435502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260927 07:07:05.436200 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:07:05.438694 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.436497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.21.2.126:35625)
0927 07:07:05.436533 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.436535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.436633 (+ 98us) server_negotiation.cc:407] Connection header received
0927 07:07:05.436677 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.436678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.436702 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.436761 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:05.436837 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.437262 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.437954 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.438130 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.438178 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.438338 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:05.438429 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:05.438503 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.438579 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.438600 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:07:05.439008 3363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:05.440696 3363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:05.442960 3485 tablet_service.cc:1467] Processing CreateTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 (DEFAULT_TABLE table=test-table [id=afd14bb4a2c846caab4a93412f1642f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:05.443171 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d367c4e36068437fbe36f3b816b1a5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:05.444765 3443 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Bootstrap starting.
I20260927 07:07:05.445096 3443 tablet_bootstrap.cc:654] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:05.445765 3443 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: No bootstrap required, opened a new log
I20260927 07:07:05.445832 3443 ts_tablet_manager.cc:1397] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:07:05.445943 3443 raft_consensus.cc:348] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.446029 3443 raft_consensus.cc:374] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:05.446082 3443 raft_consensus.cc:729] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ab5e411f624456abf30e4851b27679, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.446126 3443 consensus_queue.cc:260] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.446228 3443 ts_tablet_manager.cc:1428] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.447546 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.442649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:39487 (local address 127.0.0.1:38368)
0927 07:07:05.442823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.442834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.442885 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.443180 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.443181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.443183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.443245 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.443247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.444453 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.444454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.445431 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.445434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.447314 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.447315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.447355 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.447399 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.447447 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.447491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":123,"threads_started":1}
I20260927 07:07:05.447594 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.442845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.21.2.67:39487)
0927 07:07:05.443018 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.443022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.443034 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:07:05.443077 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.443079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.443096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.443183 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.443317 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.444355 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.445550 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.446276 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.446381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.446449 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.447465 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.447514 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260927 07:07:05.448352 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.442584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:33407 (local address 127.0.0.1:57898)
0927 07:07:05.442678 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.442690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.442743 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.445037 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.445039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.445040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.445080 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.445082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.446362 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.446364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.447236 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.447239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.448180 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.448180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.448232 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.448234 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.448275 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.448306 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 07:07:05.448410 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.442737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.21.2.66:33407)
0927 07:07:05.444855 (+ 2118us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.444860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.444871 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:07:05.444918 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.444920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.444935 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.445041 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.445126 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.446243 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.447311 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.448111 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.448184 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.448237 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.448287 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.448326 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2073,"thread_start_us":120,"threads_started":1}
I20260927 07:07:05.448447 3755 tablet_service.cc:1467] Processing CreateTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 (DEFAULT_TABLE table=test-table [id=afd14bb4a2c846caab4a93412f1642f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:05.448818 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d367c4e36068437fbe36f3b816b1a5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:05.449038 3623 tablet_service.cc:1467] Processing CreateTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 (DEFAULT_TABLE table=test-table [id=afd14bb4a2c846caab4a93412f1642f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:05.449330 3623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d367c4e36068437fbe36f3b816b1a5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:05.451682 3831 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Bootstrap starting.
I20260927 07:07:05.452286 3832 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Bootstrap starting.
I20260927 07:07:05.452430 3831 tablet_bootstrap.cc:654] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:05.452872 3832 tablet_bootstrap.cc:654] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:05.452899 3831 log.cc:824] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.453363 3832 log.cc:824] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.453632 3831 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: No bootstrap required, opened a new log
I20260927 07:07:05.453711 3831 ts_tablet_manager.cc:1397] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:05.454016 3832 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: No bootstrap required, opened a new log
I20260927 07:07:05.454068 3832 ts_tablet_manager.cc:1397] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:07:05.455302 3831 raft_consensus.cc:348] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.455516 3831 raft_consensus.cc:374] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:05.455565 3831 raft_consensus.cc:729] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e5955901d4337ae52de9ebe207d23, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.455688 3831 consensus_queue.cc:260] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.455814 3832 raft_consensus.cc:348] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.455971 3831 ts_tablet_manager.cc:1428] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:07:05.455986 3832 raft_consensus.cc:374] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:05.455997 3821 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
I20260927 07:07:05.456029 3832 raft_consensus.cc:729] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d47d5d7132450a8bbe5195d394708a, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.456144 3832 consensus_queue.cc:260] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.456404 3832 ts_tablet_manager.cc:1428] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:07:05.456532 3689 heartbeater.cc:503] Master 127.21.2.126:35625 was elected leader, sending a full tablet report...
W20260927 07:07:05.540110 3690 tablet.cc:2369] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:05.644578 3835 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:05.644757 3835 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.645202 3835 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:05.649207 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.645595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:33407 (local address 127.21.2.67:54089)
0927 07:07:05.645795 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.645807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.645884 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.646011 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.646011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.646012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.646054 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.646054 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.646846 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.646847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.648007 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.648010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.648937 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.649006 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.649010 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.649066 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.649116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
I20260927 07:07:05.649225 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.645510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:54089 (local address 127.21.2.66:33407)
0927 07:07:05.645551 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.645553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.645897 (+ 344us) server_negotiation.cc:407] Connection header received
0927 07:07:05.645941 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.645942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.645954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.645998 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.646091 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.646763 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648136 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.648806 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648942 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.648955 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.649093 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.649148 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 07:07:05.649236 3552 tablet.cc:2369] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:05.649513 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.645432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.21.2.67:44819)
0927 07:07:05.645492 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.645504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.645550 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.645735 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.645736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.645738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.645824 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.645826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.646768 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.646769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.648007 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.648010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.648916 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.649008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.649010 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.649066 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.649116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":18}
I20260927 07:07:05.649677 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.645476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:44819 (local address 127.21.2.65:42893)
0927 07:07:05.645565 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.645567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.645577 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:07:05.645612 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.645614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.645628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.645706 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.645912 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.646680 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648135 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.648817 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.648903 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.648919 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.649146 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.649196 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:07:05.649725 3643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d367c4e36068437fbe36f3b816b1a5f3" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d47d5d7132450a8bbe5195d394708a" is_pre_election: true
I20260927 07:07:05.649926 3643 raft_consensus.cc:2454] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 0.
I20260927 07:07:05.650224 3708 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a; no voters:
I20260927 07:07:05.650370 3835 raft_consensus.cc:2781] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:05.650436 3835 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:05.650470 3835 raft_consensus.cc:3037] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.650023 3505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d367c4e36068437fbe36f3b816b1a5f3" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ab5e411f624456abf30e4851b27679" is_pre_election: true
I20260927 07:07:05.650768 3505 raft_consensus.cc:2454] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 0.
I20260927 07:07:05.651345 3835 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.651537 3835 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 election: Requested vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893)
I20260927 07:07:05.651688 3643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d367c4e36068437fbe36f3b816b1a5f3" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d47d5d7132450a8bbe5195d394708a"
I20260927 07:07:05.651713 3505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d367c4e36068437fbe36f3b816b1a5f3" candidate_uuid: "351e5955901d4337ae52de9ebe207d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ab5e411f624456abf30e4851b27679"
I20260927 07:07:05.651778 3505 raft_consensus.cc:3037] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.651779 3643 raft_consensus.cc:3037] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.652628 3505 raft_consensus.cc:2454] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 1.
I20260927 07:07:05.652628 3643 raft_consensus.cc:2454] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351e5955901d4337ae52de9ebe207d23 in term 1.
I20260927 07:07:05.652802 3711 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5955901d4337ae52de9ebe207d23, b1ab5e411f624456abf30e4851b27679; no voters:
I20260927 07:07:05.652931 3835 raft_consensus.cc:2781] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:05.653113 3835 raft_consensus.cc:686] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 1 LEADER]: Becoming Leader. State: Replica: 351e5955901d4337ae52de9ebe207d23, State: Running, Role: LEADER
I20260927 07:07:05.653221 3835 consensus_queue.cc:237] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.654035 3362 catalog_manager.cc:5495] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 351e5955901d4337ae52de9ebe207d23 (127.21.2.67). New cstate: current_term: 1 leader_uuid: "351e5955901d4337ae52de9ebe207d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:05.672606 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.670458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.21.2.67:39487)
0927 07:07:05.670528 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.670529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.670537 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:05.670569 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.670569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.670588 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.670646 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:05.670756 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.671214 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.671954 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.672131 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.672181 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.672313 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:05.672404 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:05.672464 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.672492 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.672510 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 07:07:05.672706 3822 tablet.cc:2369] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:05.674273 3643 raft_consensus.cc:1261] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Refusing update from remote peer 351e5955901d4337ae52de9ebe207d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:05.674275 3505 raft_consensus.cc:1261] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Refusing update from remote peer 351e5955901d4337ae52de9ebe207d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:05.674630 3838 consensus_queue.cc:1035] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:05.674767 3835 consensus_queue.cc:1035] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:05.676697 3841 mvcc.cc:205] Tried to move back new op lower bound from 7333858613959208960 to 7333858613876289536. Current Snapshot: MvccSnapshot[applied={T|T < 7333858613959208960}]
I20260927 07:07:05.677170 3840 mvcc.cc:205] Tried to move back new op lower bound from 7333858613959208960 to 7333858613876289536. Current Snapshot: MvccSnapshot[applied={T|T < 7333858613959208960}]
I20260927 07:07:05.677599 3842 mvcc.cc:205] Tried to move back new op lower bound from 7333858613959208960 to 7333858613876289536. Current Snapshot: MvccSnapshot[applied={T|T < 7333858613959208960}]
I20260927 07:07:05.679057 3363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:05.679157 3363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:05.679914 3363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Sending DeleteTablet for 3 replicas of tablet d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:05.680244 3485 tablet_service.cc:1514] Processing DeleteTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48702
I20260927 07:07:05.680270 3623 tablet_service.cc:1514] Processing DeleteTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:57898
I20260927 07:07:05.680332 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3401
I20260927 07:07:05.680475 3755 tablet_service.cc:1514] Processing DeleteTablet for tablet d367c4e36068437fbe36f3b816b1a5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:38368
I20260927 07:07:05.680922 3850 tablet_replica.cc:330] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: stopping tablet replica
I20260927 07:07:05.681017 3850 raft_consensus.cc:2229] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:05.681100 3850 raft_consensus.cc:2258] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.681523 3850 ts_tablet_manager.cc:1905] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.682508 3851 tablet_replica.cc:330] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: stopping tablet replica
I20260927 07:07:05.682624 3851 raft_consensus.cc:2229] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:05.682717 3851 raft_consensus.cc:2258] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.683096 3851 ts_tablet_manager.cc:1905] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.683501 3850 ts_tablet_manager.cc:1918] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.683579 3850 log.cc:1196] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:05.683889 3850 ts_tablet_manager.cc:1939] T d367c4e36068437fbe36f3b816b1a5f3 P 61d47d5d7132450a8bbe5195d394708a: Deleting consensus metadata
I20260927 07:07:05.684409 3347 catalog_manager.cc:4841] TS 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): tablet d367c4e36068437fbe36f3b816b1a5f3 (table test-table [id=afd14bb4a2c846caab4a93412f1642f4]) successfully deleted
I20260927 07:07:05.684705 3851 ts_tablet_manager.cc:1918] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.684783 3851 log.cc:1196] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:05.685019 3851 ts_tablet_manager.cc:1939] T d367c4e36068437fbe36f3b816b1a5f3 P 351e5955901d4337ae52de9ebe207d23: Deleting consensus metadata
I20260927 07:07:05.685413 3349 catalog_manager.cc:4841] TS 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): tablet d367c4e36068437fbe36f3b816b1a5f3 (table test-table [id=afd14bb4a2c846caab4a93412f1642f4]) successfully deleted
W20260927 07:07:05.690577 3350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:42893 (error 108)
I20260927 07:07:05.690853 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3556
I20260927 07:07:05.691661 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.691528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:42893 (local address 127.0.0.1:59704)
0927 07:07:05.691578 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.691616 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:07:05.691802 3350 catalog_manager.cc:4584] TS b1ab5e411f624456abf30e4851b27679 (127.21.2.65:42893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d367c4e36068437fbe36f3b816b1a5f3: Network error: Client connection negotiation failed: client connection to 127.21.2.65:42893: connect: Connection refused (error 111)
I20260927 07:07:05.696997 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3692
I20260927 07:07:05.701898 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3330
I20260927 07:07:05.707193 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--webserver_interface=127.21.2.126
--webserver_port=37885
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:05.788412 3853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:05.788617 3853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:05.788640 3853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:05.789829 3853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:05.789876 3853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:05.789891 3853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:05.789903 3853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:05.789934 3853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:05.791527 3853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:05.791788 3853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:05.792037 3853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:07:05.797044 3853 server_base.cc:1034] running on GCE node
W20260927 07:07:05.797017 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.797019 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.797006 3858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.797468 3853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:05.797765 3853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:05.798933 3853 hybrid_clock.cc:648] HybridClock initialized: now 1790492825798928 us; error 38 us; skew 500 ppm
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:05.801368 3853 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:05.801862 3853 webserver.cc:460] Webserver started at http://127.21.2.126:37885/ using document root <none> and password file <none>
I20260927 07:07:05.802106 3853 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:05.802179 3853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:05.803602 3853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.804282 3868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.804445 3853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:07:05.804551 3853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "acae0a2eb5624f868fc665d6cf5c9cd8"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "051196b81573a259c6565d36236b1d97"
server_key_iv: "58b5e40c645624896da46e25bc2ef110"
server_key_version: "encryptionkey@0"
I20260927 07:07:05.804913 3853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:05.817699 3853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:05.819577 3853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:05.819753 3853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.823709 3853 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:35625
I20260927 07:07:05.823760 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:35625 every 8 connection(s)
I20260927 07:07:05.824152 3853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:05.825706 3921 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:05.826372 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap starting.
I20260927 07:07:05.832178 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3853
I20260927 07:07:05.832492 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:42893
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=39101
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:05.834870 3921 log.cc:824] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.837260 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:05.837574 3921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8: Bootstrap complete.
I20260927 07:07:05.839919 3921 raft_consensus.cc:348] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.840169 3921 raft_consensus.cc:729] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.840355 3921 consensus_queue.cc:260] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.840435 3921 raft_consensus.cc:388] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:05.840488 3921 raft_consensus.cc:482] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:05.840541 3921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:07:05.841517 3921 raft_consensus.cc:504] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.841636 3921 leader_election.cc:302] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acae0a2eb5624f868fc665d6cf5c9cd8; no voters:
I20260927 07:07:05.841847 3921 leader_election.cc:288] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:07:05.841885 3925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:07:05.841990 3925 raft_consensus.cc:686] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [term 3 LEADER]: Becoming Leader. State: Replica: acae0a2eb5624f868fc665d6cf5c9cd8, State: Running, Role: LEADER
I20260927 07:07:05.842077 3925 consensus_queue.cc:237] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } }
I20260927 07:07:05.842140 3921 sys_catalog.cc:564] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:05.843472 3927 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:05.843518 3926 sys_catalog.cc:455] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acae0a2eb5624f868fc665d6cf5c9cd8. Latest consensus state: current_term: 3 leader_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acae0a2eb5624f868fc665d6cf5c9cd8" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 35625 } } }
I20260927 07:07:05.843575 3927 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:05.843608 3926 sys_catalog.cc:458] T 00000000000000000000000000000000 P acae0a2eb5624f868fc665d6cf5c9cd8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:05.844013 3936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:05.844818 3936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:05.844928 3936 catalog_manager.cc:1259] Loaded cluster ID: 27d59a5127854684a180f995020ec739
I20260927 07:07:05.844962 3936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 07:07:05.845168 3936 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdaa4df69ad google::LogMessage::Fail() at ??:0
@ 0x7fdaa4df988a google::LogMessage::SendToLog() at ??:0
@ 0x7fdaa4df64de google::LogMessage::Flush() at ??:0
@ 0x7fdaa4df7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdaacd170f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdaacd208b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdaa6cc40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdaa6cb9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdaa7b1d6db start_thread at ??:0
@ 0x7fdaa38f461f clone at ??:0
W20260927 07:07:05.924944 3923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:05.925189 3923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:05.925232 3923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:05.926412 3923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:05.926481 3923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:05.926528 3923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:07:05.928133 3923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:42893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=39101
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:05.928416 3923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:05.928644 3923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:05.933943 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.933933 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:05.933952 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:05.934202 3923 server_base.cc:1034] running on GCE node
I20260927 07:07:05.934402 3923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:05.934667 3923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:05.935869 3923 hybrid_clock.cc:648] HybridClock initialized: now 1790492825935840 us; error 40 us; skew 500 ppm
Sep 27 07:07:05 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:05.938630 3923 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:07:05.939100 3923 webserver.cc:460] Webserver started at http://127.21.2.65:39101/ using document root <none> and password file <none>
I20260927 07:07:05.939344 3923 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:05.939448 3923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:05.940886 3923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.941658 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:05.941891 3923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:05.941980 3923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "b1ab5e411f624456abf30e4851b27679"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "18e87efda9d01120234e57c8518fb58e"
server_key_iv: "938c86183e65d745990cb5959f60371e"
server_key_version: "encryptionkey@0"
I20260927 07:07:05.942466 3923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:05.962600 3923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:05.966311 3923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:05.966529 3923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:05.966898 3923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:05.967537 3974 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:07:05.969024 3923 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:07:05.969084 3923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:07:05.969122 3923 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:07:05.970042 3923 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:07:05.970089 3923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:07:05.970167 3974 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Bootstrap starting.
I20260927 07:07:05.977725 3923 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:42893
I20260927 07:07:05.978298 3923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:05.978307 4081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:42893 every 8 connection(s)
I20260927 07:07:05.980341 3974 log.cc:824] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:05.982398 3974 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:05.982695 3974 tablet_bootstrap.cc:492] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Bootstrap complete.
I20260927 07:07:05.983667 3974 ts_tablet_manager.cc:1397] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 07:07:05.984638 3974 raft_consensus.cc:348] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.984982 3974 raft_consensus.cc:729] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ab5e411f624456abf30e4851b27679, State: Initialized, Role: FOLLOWER
I20260927 07:07:05.985152 3974 consensus_queue.cc:260] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:05.985409 3974 ts_tablet_manager.cc:1428] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:07:05.988384 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 3923
I20260927 07:07:05.988688 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:33407
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=34815
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:06.110455 4087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:06.110657 4087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:06.110680 4087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:06.111903 4087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:06.111945 4087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:06.111971 4087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:07:06.113407 4087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:33407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=34815
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:06.113677 4087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260927 07:07:06.113777 4084 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:05.979172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:46705)
0927 07:07:05.979401 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.979468 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.980132 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:06.113523 (+133391us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260927 07:07:06.113969 4087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:06.114037 4082 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:35625 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:35625: Client connection negotiation failed: client connection to 127.21.2.126:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:07:06.114416 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.114278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:56395)
0927 07:07:06.114338 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.114389 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:07:06.114812 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.114711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:45339)
0927 07:07:06.114759 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.114789 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:07:06.114926 4082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:07:06.118991 4092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:06.118999 4093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:06.119050 4095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:06.119433 4087 server_base.cc:1034] running on GCE node
I20260927 07:07:06.119647 4087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:06.119887 4087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:06.121049 4087 hybrid_clock.cc:648] HybridClock initialized: now 1790492826121040 us; error 37 us; skew 500 ppm
Sep 27 07:07:06 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492826, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:06.123625 4087 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:07:06.124049 4087 webserver.cc:460] Webserver started at http://127.21.2.66:34815/ using document root <none> and password file <none>
I20260927 07:07:06.124284 4087 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:06.124353 4087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:06.125902 4087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:06.126487 4102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.126659 4087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:07:06.126755 4087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "61d47d5d7132450a8bbe5195d394708a"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "010eb1a1ae2ee728fa25ed6a2ca81237"
server_key_iv: "6c9bbebaf91b6a5c54826fdff48cc36c"
server_key_version: "encryptionkey@0"
I20260927 07:07:06.127099 4087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:06.141990 4087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:06.144416 4087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:06.144587 4087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:06.144873 4087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:06.145190 4087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:06.145243 4087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.145287 4087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:06.145313 4087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.151352 4087 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:33407
I20260927 07:07:06.151433 4215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:33407 every 8 connection(s)
I20260927 07:07:06.151810 4087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:06.153043 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.152470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:38793)
0927 07:07:06.152621 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.152703 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260927 07:07:06.153254 4216 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:35625 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:35625: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
I20260927 07:07:06.153532 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.153438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:33235)
0927 07:07:06.153486 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.153510 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 07:07:06.153807 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.153723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:42497)
0927 07:07:06.153765 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.153786 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 07:07:06.153913 4216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:07:06.155027 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4087
I20260927 07:07:06.155234 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:39487
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:35625
--webserver_port=40797
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:06.235339 4219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:06.235617 4219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:06.235641 4219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:06.236752 4219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:06.236788 4219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:06.236812 4219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:07:06.238157 4219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:39487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=40797
--tserver_master_addrs=127.21.2.126:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:06.238327 4219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:06.238493 4219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:06.243414 4225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:06.243427 4227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:06.243424 4224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:06.243662 4219 server_base.cc:1034] running on GCE node
I20260927 07:07:06.243889 4219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:06.244153 4219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:06.245282 4219 hybrid_clock.cc:648] HybridClock initialized: now 1790492826245269 us; error 39 us; skew 500 ppm
Sep 27 07:07:06 dist-test-slave-ltqz krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492826, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:06.247924 4219 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:07:06.248356 4219 webserver.cc:460] Webserver started at http://127.21.2.67:40797/ using document root <none> and password file <none>
I20260927 07:07:06.248580 4219 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:06.248626 4219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:06.249897 4219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:07:06.250422 4234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.250571 4219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.250672 4219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "351e5955901d4337ae52de9ebe207d23"
format_stamp: "Formatted at 2026-09-27 07:07:04 on dist-test-slave-ltqz"
server_key: "440a985df2e11b7139a2ff6c454867f5"
server_key_iv: "919a2d6d6efa6f6036197636b55cd303"
server_key_version: "encryptionkey@0"
I20260927 07:07:06.251022 4219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:06.275236 4219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:06.277154 4219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:06.277336 4219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:06.277592 4219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:06.277922 4219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:06.277978 4219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.278019 4219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:06.278048 4219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:06.283763 4219 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:39487
I20260927 07:07:06.283808 4347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:39487 every 8 connection(s)
I20260927 07:07:06.284137 4219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:06.285151 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.284795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:43211)
0927 07:07:06.284939 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.285018 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
W20260927 07:07:06.285619 4348 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:35625 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:35625: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
I20260927 07:07:06.285914 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.285826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:58395)
0927 07:07:06.285871 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.285897 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 07:07:06.286142 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.286095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:42147)
0927 07:07:06.286113 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.286130 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 07:07:06.286252 4348 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:07:06.290364 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4219
I20260927 07:07:07.115584 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.115284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:49661)
0927 07:07:07.115468 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.115506 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
I20260927 07:07:07.154479 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.154230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:41799)
0927 07:07:07.154352 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.154389 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260927 07:07:07.286768 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.286570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:51913)
0927 07:07:07.286684 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.286719 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260927 07:07:07.787048 4355 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:07.787190 4355 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:07.787601 4355 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
Sep 27 07:07:07 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.67@KRBTEST.COM
Sep 27 07:07:07 dist-test-slave-ltqz krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492825, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.66@KRBTEST.COM
I20260927 07:07:07.801535 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.787758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:33407 (local address 127.21.2.65:43895)
0927 07:07:07.787909 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.787923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.787989 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.788748 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.788751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.789162 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:07.789404 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.789406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.790620 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.790621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.791428 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.791430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.792021 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.792022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.792141 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.796208 (+ 4067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:07.796224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:07.798101 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:07.800136 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:07.800137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:07.800342 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:07.800581 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:07.800583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:07.800656 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:07.801032 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:07.801033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:07.801139 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.801248 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.801433 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":39,"thread_start_us":53,"threads_started":1}
I20260927 07:07:07.801548 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.787868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:43895 (local address 127.21.2.66:33407)
0927 07:07:07.788055 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.788064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.788085 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:07:07.788507 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.788509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.788549 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.788721 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:07.789505 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.790506 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.791522 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.791913 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.792037 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.798233 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:07.798240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:07.798240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:07.798252 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:07.800049 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800412 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.800414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.800415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.800506 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800724 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.800725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.800725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.800942 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:07.801037 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.801300 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.801422 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":56,"threads_started":1}
I20260927 07:07:07.801576 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.787888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:37087 (local address 127.21.2.67:39487)
0927 07:07:07.788114 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.788122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.788140 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:07:07.788539 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.788541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.788575 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.788747 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:07.789566 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.790493 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.791510 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.791912 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.792048 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.797922 (+ 5874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:07.797927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:07.797928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:07.797941 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:07.799691 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800322 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.800326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.800326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.800437 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800678 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.800679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.800680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.800923 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:07.801022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.801308 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.801462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":48,"threads_started":1}
I20260927 07:07:07.801659 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.787798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:39487 (local address 127.21.2.65:37087)
0927 07:07:07.787923 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.787932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.787989 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.788750 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.788753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.789166 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:07.789375 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.789382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.790598 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.790599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.791415 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.791417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.792026 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.792027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.792142 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.796234 (+ 4092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:07.796239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:07.797761 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:07.799778 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:07.799785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:07.799787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:07.800223 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:07.800490 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:07.800491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:07.800491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:07.800592 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:07.801002 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:07.801009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:07.801141 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.801248 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.801513 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":156,"thread_start_us":77,"threads_started":1}
W20260927 07:07:07.802688 3968 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
W20260927 07:07:07.802786 3970 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:07.802846 3970 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1ab5e411f624456abf30e4851b27679; no voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a
I20260927 07:07:07.802999 4355 raft_consensus.cc:2726] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:08.116137 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:08.115995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:34281)
0927 07:07:08.116061 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:08.116098 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:07:08.155076 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:08.154868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:36387)
0927 07:07:08.154991 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:08.155036 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260927 07:07:08.287525 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:08.287230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:39419)
0927 07:07:08.287419 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:08.287482 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260927 07:07:09.116804 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:09.116589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:54891)
0927 07:07:09.116721 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:09.116758 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260927 07:07:09.155738 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:09.155551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:55605)
0927 07:07:09.155665 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:09.155699 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 07:07:09.288220 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:09.287989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:33441)
0927 07:07:09.288142 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:09.288176 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260927 07:07:09.810801 4365 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:09.810890 4365 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:09.811095 4365 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
W20260927 07:07:09.811638 3968 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
W20260927 07:07:09.811719 3970 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:09.811813 3970 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1ab5e411f624456abf30e4851b27679; no voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a
I20260927 07:07:09.811903 4365 raft_consensus.cc:2726] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:10.117563 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.117310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:38609)
0927 07:07:10.117462 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.117503 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260927 07:07:10.156425 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.156176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:41961)
0927 07:07:10.156295 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.156336 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":99,"threads_started":1}
I20260927 07:07:10.289014 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.288741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:42785)
0927 07:07:10.288900 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.288954 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260927 07:07:11.118345 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.118146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:59423)
0927 07:07:11.118269 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.118300 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260927 07:07:11.157075 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.156869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:39645)
0927 07:07:11.156995 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.157035 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260927 07:07:11.289752 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.289524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:52493)
0927 07:07:11.289661 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.289704 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260927 07:07:11.709151 4372 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:11.709271 4372 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:11.709512 4372 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
W20260927 07:07:11.709926 3968 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
W20260927 07:07:11.710023 3970 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:11.710101 3970 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1ab5e411f624456abf30e4851b27679; no voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a
I20260927 07:07:11.710177 4372 raft_consensus.cc:2726] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:12.119062 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.118830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:43727)
0927 07:07:12.118967 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.119004 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260927 07:07:12.157809 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.157531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:59471)
0927 07:07:12.157729 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.157769 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":56,"threads_started":1}
I20260927 07:07:12.290448 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.290233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:38745)
0927 07:07:12.290368 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.290407 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260927 07:07:13.119922 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.119691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:53987)
0927 07:07:13.119831 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.119872 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260927 07:07:13.158535 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.158293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:36903)
0927 07:07:13.158457 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.158489 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260927 07:07:13.291118 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.290888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:57709)
0927 07:07:13.291035 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.291076 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260927 07:07:14.086486 4379 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:14.086603 4379 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:14.086817 4379 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
W20260927 07:07:14.087153 3968 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
W20260927 07:07:14.087239 3970 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:14.087270 3970 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1ab5e411f624456abf30e4851b27679; no voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a
I20260927 07:07:14.087383 4379 raft_consensus.cc:2726] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:14.120687 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.120428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:37633)
0927 07:07:14.120601 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:14.120637 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260927 07:07:14.159301 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.159103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:57453)
0927 07:07:14.159223 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:14.159259 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260927 07:07:14.291817 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.291623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:45233)
0927 07:07:14.291744 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:14.291778 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260927 07:07:15.121379 4383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:15.121179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:58763)
0927 07:07:15.121300 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:15.121334 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 07:07:15.160041 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:15.159813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:38187)
0927 07:07:15.159960 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:15.159995 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260927 07:07:15.292591 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:15.292289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:33053)
0927 07:07:15.292487 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:15.292539 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260927 07:07:16.122138 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:16.121892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:35597)
0927 07:07:16.122051 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:16.122090 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260927 07:07:16.160720 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:16.160488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:46135)
0927 07:07:16.160642 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:16.160673 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260927 07:07:16.293404 4388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:16.293118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:50905)
0927 07:07:16.293260 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:16.293302 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260927 07:07:17.123703 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:17.122924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:43885)
0927 07:07:17.123148 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:17.123630 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260927 07:07:17.161373 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:17.161153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:32849)
0927 07:07:17.161297 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:17.161328 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260927 07:07:17.294256 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:17.294008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:41379)
0927 07:07:17.294166 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:17.294209 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260927 07:07:18.124500 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:18.124245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:49901)
0927 07:07:18.124412 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:18.124454 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260927 07:07:18.162138 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:18.161845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:46487)
0927 07:07:18.162006 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:18.162078 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260927 07:07:18.294958 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:18.294735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:50259)
0927 07:07:18.294874 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:18.294914 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260927 07:07:19.125483 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.125056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:54523)
0927 07:07:19.125329 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:19.125404 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":63,"threads_started":1}
I20260927 07:07:19.162791 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.162581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:53869)
0927 07:07:19.162714 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:19.162747 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260927 07:07:19.295753 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.295528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:46865)
0927 07:07:19.295662 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:19.295705 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260927 07:07:19.537794 4398 raft_consensus.cc:482] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:19.537899 4398 raft_consensus.cc:504] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d47d5d7132450a8bbe5195d394708a" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 33407 } } peers { permanent_uuid: "351e5955901d4337ae52de9ebe207d23" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 39487 } } peers { permanent_uuid: "b1ab5e411f624456abf30e4851b27679" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 42893 } }
I20260927 07:07:19.538164 4398 leader_election.cc:288] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407), 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487)
W20260927 07:07:19.538565 3970 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 351e5955901d4337ae52de9ebe207d23 (127.21.2.67:39487): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
W20260927 07:07:19.538640 3968 leader_election.cc:341] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61d47d5d7132450a8bbe5195d394708a (127.21.2.66:33407): Not found: Tablet not found: d367c4e36068437fbe36f3b816b1a5f3
I20260927 07:07:19.538713 3968 leader_election.cc:302] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1ab5e411f624456abf30e4851b27679; no voters: 351e5955901d4337ae52de9ebe207d23, 61d47d5d7132450a8bbe5195d394708a
I20260927 07:07:19.538822 4398 raft_consensus.cc:2726] T d367c4e36068437fbe36f3b816b1a5f3 P b1ab5e411f624456abf30e4851b27679 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:20.126209 4399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.125987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:51599)
0927 07:07:20.126122 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:20.126161 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260927 07:07:20.163690 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.163231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:45621)
0927 07:07:20.163355 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:20.163643 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260927 07:07:20.296471 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.296247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:48493)
0927 07:07:20.296386 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:20.296427 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260927 07:07:21.126987 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:21.126782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.65:53515)
0927 07:07:21.126907 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:21.126942 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260927 07:07:21.164340 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:21.164153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.66:34511)
0927 07:07:21.164266 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:21.164297 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 07:07:21.297350 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:21.297049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:35625 (local address 127.21.2.67:48633)
0927 07:07:21.297250 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:21.297302 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260927 07:07:21.300721 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 3923
I20260927 07:07:21.305603 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4087
I20260927 07:07:21.309999 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4219
2026-09-27T07:07:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18439 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:21 dist-test-slave-ltqz krb5kdc[4407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:21 dist-test-slave-ltqz krb5kdc[4407](info): set up 2 sockets
Sep 27 07:07:21 dist-test-slave-ltqz krb5kdc[4407](info): commencing operation
krb5kdc: starting...
W20260927 07:07:22.355770 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:07:22.470393 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:22.551254 4420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:22.551640 4420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:22.551682 4420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:22.552938 4420 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:07:22.553016 4420 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:07:22.553040 4420 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:07:22.553058 4420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:22.553092 4420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:22.553118 4420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:22.553145 4420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:22.553176 4420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:22.554860 4420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:22.555162 4420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:22.555433 4420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:22.560393 4426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.560411 4425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.560529 4420 server_base.cc:1034] running on GCE node
W20260927 07:07:22.560401 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.561004 4420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:22.561321 4420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:22.562487 4420 hybrid_clock.cc:648] HybridClock initialized: now 1790492842562487 us; error 43 us; skew 500 ppm
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:22.565053 4420 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:22.565491 4420 webserver.cc:460] Webserver started at http://127.21.2.126:38237/ using document root <none> and password file <none>
I20260927 07:07:22.565732 4420 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:22.565801 4420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:22.565955 4420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:22.567013 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "441dc8a7f29e4ae181d594a0318550d6"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "d2ebc4f42b361cfb8f43e8b2ca5f526a"
server_key_iv: "a281e4e37b0228f044af9ca3e057d476"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.567435 4420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "441dc8a7f29e4ae181d594a0318550d6"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "d2ebc4f42b361cfb8f43e8b2ca5f526a"
server_key_iv: "a281e4e37b0228f044af9ca3e057d476"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.568768 4420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:22.569439 4435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.569602 4420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.569696 4420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "441dc8a7f29e4ae181d594a0318550d6"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "d2ebc4f42b361cfb8f43e8b2ca5f526a"
server_key_iv: "a281e4e37b0228f044af9ca3e057d476"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.569784 4420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:22.586536 4420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:22.588483 4420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:22.588651 4420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:22.592505 4420 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40959
I20260927 07:07:22.592557 4487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40959 every 8 connection(s)
I20260927 07:07:22.592924 4420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:22.593793 4488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:22.595067 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4420
I20260927 07:07:22.595166 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:07:22.595418 21513 external_mini_cluster.cc:1428] Setting key f8c1eede011c36d1a569c298e0757840
I20260927 07:07:22.596504 4488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap starting.
I20260927 07:07:22.597234 4488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:22.597582 4488 log.cc:824] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:22.598280 4488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: No bootstrap required, opened a new log
I20260927 07:07:22.600612 4488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:22.600803 4488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:22.600842 4488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Initialized, Role: FOLLOWER
I20260927 07:07:22.600975 4488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:22.601059 4488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:22.601095 4488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:22.601150 4488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:22.601890 4488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:22.601979 4488 leader_election.cc:302] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441dc8a7f29e4ae181d594a0318550d6; no voters:
I20260927 07:07:22.602164 4488 leader_election.cc:288] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:22.602202 4493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:22.602382 4493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 LEADER]: Becoming Leader. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Running, Role: LEADER
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:22.602463 4488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:22.602527 4493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:22.602907 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441dc8a7f29e4ae181d594a0318550d6. Latest consensus state: current_term: 1 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:07:22.603008 4494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:07:22.603104 4494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:22.603057 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:22.603448 4500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:22.604538 4500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:22.606139 4500 catalog_manager.cc:1347] Generated new cluster ID: edd8cc8326114f4c8554b297155ab5fa
I20260927 07:07:22.606195 4500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:22.606841 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.595939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.21.2.126:40959)
0927 07:07:22.596121 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:22.596129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:22.596162 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:07:22.596538 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:22.596544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:22.596627 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:22.596761 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:22.597529 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.598609 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.599683 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.600064 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.600142 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.603026 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:22.603031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:22.603031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:22.603043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:22.605134 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.605544 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.605548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.605548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.605662 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.605897 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.605898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.605898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.606232 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:22.606418 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:22.606536 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 07:07:22.606679 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260927 07:07:22.641009 4500 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:22.641764 4500 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:22.651660 4500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Generated new TSK 0
I20260927 07:07:22.651907 4500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:07:22.677927 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:22.759776 4514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:22.759979 4514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:22.760003 4514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:22.761128 4514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:22.761166 4514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:22.761193 4514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:07:22.762660 4514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:22.762862 4514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:22.763044 4514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:22.768113 4519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.768114 4520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.768167 4514 server_base.cc:1034] running on GCE node
W20260927 07:07:22.768110 4522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.768498 4514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:22.768759 4514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:22.769948 4514 hybrid_clock.cc:648] HybridClock initialized: now 1790492842769926 us; error 35 us; skew 500 ppm
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:22.772748 4514 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:07:22.773170 4514 webserver.cc:460] Webserver started at http://127.21.2.65:43433/ using document root <none> and password file <none>
I20260927 07:07:22.773415 4514 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:22.773483 4514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:22.773671 4514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:22.774704 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "e8b8d64567c9f98cc3a5969d8c355db0"
server_key_iv: "3ba08a02ca74492e37c35886403693bc"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.775054 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "e8b8d64567c9f98cc3a5969d8c355db0"
server_key_iv: "3ba08a02ca74492e37c35886403693bc"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.776449 4514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:07:22.777240 4529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.777415 4514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:22.777513 4514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "e8b8d64567c9f98cc3a5969d8c355db0"
server_key_iv: "3ba08a02ca74492e37c35886403693bc"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.777598 4514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:22.794585 4514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:22.796561 4514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:22.796738 4514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:22.796975 4514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:22.797273 4514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:22.797326 4514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.797366 4514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:22.797394 4514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.803462 4514 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:36743
I20260927 07:07:22.803550 4642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:36743 every 8 connection(s)
I20260927 07:07:22.803848 4514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:22.812911 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4514
I20260927 07:07:22.812923 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.804576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:53775 (local address 127.21.2.126:40959)
0927 07:07:22.804640 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:22.804642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:22.804854 (+ 212us) server_negotiation.cc:407] Connection header received
0927 07:07:22.805303 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:22.805304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:22.805313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:22.805366 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:22.806251 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.806748 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.807787 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.807998 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.808049 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.810028 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:22.810029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:22.810029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:22.810035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:22.811532 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.811925 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.811926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.811927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.811999 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.812269 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.812271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.812271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.812392 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:22.812463 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:22.812744 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 07:07:22.812829 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:07:22.813021 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:22.813103 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.804545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:53775)
0927 07:07:22.804715 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:22.804753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:22.805208 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:22.805417 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:22.805420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:22.805976 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:22.806167 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.806172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.806826 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.806826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.807705 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.807708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.808067 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.808067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.808162 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.808618 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:22.808626 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:22.809947 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:22.811609 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:22.811617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:22.811619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:22.811844 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:22.812073 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:22.812073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:22.812074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:22.812180 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:22.812465 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:22.812472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:22.812607 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 07:07:22.812716 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:07:22.812875 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260927 07:07:22.813231 21513 external_mini_cluster.cc:1428] Setting key c292fc6f4de3d3a6e98fbcb7a61f779a
I20260927 07:07:22.813794 4643 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:22.813915 4643 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:22.814106 4643 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:22.814718 4451 ts_manager.cc:194] Registered new tserver with Master: e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743)
I20260927 07:07:22.815614 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:53775
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:07:22.836025 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:22.916985 4648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:22.917212 4648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:22.917249 4648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:22.918409 4648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:22.918478 4648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:22.918526 4648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:07:22.920063 4648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:22.920342 4648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:22.920575 4648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:22.925535 4654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.925611 4656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.925531 4653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.925800 4648 server_base.cc:1034] running on GCE node
I20260927 07:07:22.926044 4648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:22.926290 4648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:22.927438 4648 hybrid_clock.cc:648] HybridClock initialized: now 1790492842927418 us; error 29 us; skew 500 ppm
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:22.929931 4648 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:07:22.930370 4648 webserver.cc:460] Webserver started at http://127.21.2.66:40149/ using document root <none> and password file <none>
I20260927 07:07:22.930593 4648 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:22.930656 4648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:22.930801 4648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:22.931936 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "926cddd481dc4140a6926eef415d1f3b"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "fdce70011149f28a8ae4bf7b4333282d"
server_key_iv: "ca56076e33d1e0c5140d70d8c717cb6b"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.932297 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "926cddd481dc4140a6926eef415d1f3b"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "fdce70011149f28a8ae4bf7b4333282d"
server_key_iv: "ca56076e33d1e0c5140d70d8c717cb6b"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.933548 4648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:22.934232 4663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.934397 4648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:07:22.934491 4648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "926cddd481dc4140a6926eef415d1f3b"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "fdce70011149f28a8ae4bf7b4333282d"
server_key_iv: "ca56076e33d1e0c5140d70d8c717cb6b"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.934578 4648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:22.943120 4648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:22.945077 4648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:22.945240 4648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:22.945461 4648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:22.945768 4648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:22.945822 4648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.945863 4648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:22.945890 4648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.953299 4648 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46477
I20260927 07:07:22.953339 4776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46477 every 8 connection(s)
I20260927 07:07:22.953676 4648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:07:22 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:22.960877 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4648
I20260927 07:07:22.960988 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:22.961203 21513 external_mini_cluster.cc:1428] Setting key d7e45a2b3b63d8a0a0ce955169190207
I20260927 07:07:22.963032 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.954436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:45661)
0927 07:07:22.954615 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:22.954651 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:22.955132 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:22.955358 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:22.955389 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:22.955961 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:22.956168 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.956173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.956867 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.956868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.957755 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.957758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.958106 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.958106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.958206 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.958686 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:22.958694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:22.960126 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:22.961672 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:22.961678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:22.961680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:22.961978 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:22.962187 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:22.962187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:22.962188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:22.962287 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:22.962561 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:22.962566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:22.962678 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 07:07:22.962757 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:07:22.962873 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260927 07:07:22.963027 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.954446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:45661 (local address 127.21.2.126:40959)
0927 07:07:22.954520 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:22.954522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:22.954745 (+ 223us) server_negotiation.cc:407] Connection header received
0927 07:07:22.955220 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:22.955222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:22.955231 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:22.955285 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:22.956284 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.956780 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.957838 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.958020 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.958070 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.960209 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:22.960211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:22.960211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:22.960219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:22.961597 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.962051 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.962052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.962052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.962116 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:22.962352 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:22.962353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:22.962353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:22.962498 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:22.962568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:22.962800 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 07:07:22.962889 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:07:22.963549 4777 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:22.963670 4777 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:22.963835 4777 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:22.964251 4451 ts_manager.cc:194] Registered new tserver with Master: 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
I20260927 07:07:22.964643 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:45661
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:07:22.984086 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:23.065258 4782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:23.065542 4782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:23.065629 4782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:23.067345 4782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:23.067440 4782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:23.067481 4782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:07:23.069365 4782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:23.069633 4782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:23.069823 4782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:23.074738 4787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:23.074743 4790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:23.074875 4782 server_base.cc:1034] running on GCE node
W20260927 07:07:23.074749 4788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:23.075145 4782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:23.075455 4782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:23.076637 4782 hybrid_clock.cc:648] HybridClock initialized: now 1790492843076586 us; error 65 us; skew 500 ppm
Sep 27 07:07:23 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:23.079213 4782 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:07:23.079785 4782 webserver.cc:460] Webserver started at http://127.21.2.67:33607/ using document root <none> and password file <none>
I20260927 07:07:23.080052 4782 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:23.080123 4782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:23.080286 4782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:23.081434 4782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
format_stamp: "Formatted at 2026-09-27 07:07:23 on dist-test-slave-ltqz"
server_key: "47e1171263d9cd15390509f9bf783b9f"
server_key_iv: "d8c4246491633530a5763eb795e057a8"
server_key_version: "encryptionkey@0"
I20260927 07:07:23.081820 4782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
format_stamp: "Formatted at 2026-09-27 07:07:23 on dist-test-slave-ltqz"
server_key: "47e1171263d9cd15390509f9bf783b9f"
server_key_iv: "d8c4246491633530a5763eb795e057a8"
server_key_version: "encryptionkey@0"
I20260927 07:07:23.083228 4782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:07:23.084278 4797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.084702 4782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:07:23.084813 4782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
format_stamp: "Formatted at 2026-09-27 07:07:23 on dist-test-slave-ltqz"
server_key: "47e1171263d9cd15390509f9bf783b9f"
server_key_iv: "d8c4246491633530a5763eb795e057a8"
server_key_version: "encryptionkey@0"
I20260927 07:07:23.084896 4782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:23.095844 4782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:23.097664 4782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:23.097829 4782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:23.098063 4782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:23.098367 4782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:23.098400 4782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.098423 4782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:23.098467 4782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.104574 4782 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:43857
I20260927 07:07:23.104663 4910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:43857 every 8 connection(s)
I20260927 07:07:23.104956 4782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:23.108914 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4782
I20260927 07:07:23.109015 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:23.109189 21513 external_mini_cluster.cc:1428] Setting key 6dcb3d3849f3e73f132f23d3955211b5
Sep 27 07:07:23 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:23.114051 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.105710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:59027 (local address 127.21.2.126:40959)
0927 07:07:23.105751 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.105753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.105940 (+ 187us) server_negotiation.cc:407] Connection header received
0927 07:07:23.106468 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.106469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.106482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.106531 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.107563 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.108081 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.109081 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.109331 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.109398 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.111656 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:23.111657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:23.111658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:23.111664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:23.112872 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.113285 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.113286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.113286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.113332 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.113538 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.113539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.113539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.113633 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:23.113679 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.113904 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.113965 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:07:23.114248 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.105669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:59027)
0927 07:07:23.105811 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.105856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.106359 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.106598 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.106602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.107191 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:23.107458 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.107464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.108185 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.108186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.108973 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.108976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.109455 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.109457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.109615 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.110205 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:23.110214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:23.111577 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:23.112974 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:23.112988 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:23.112990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:23.113220 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:23.113396 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:23.113396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:23.113397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:23.113485 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:23.113695 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:23.113699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:23.113802 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.113879 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.113988 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260927 07:07:23.114674 4911 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:23.114773 4911 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:23.114943 4911 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:23.115300 4451 ts_manager.cc:194] Registered new tserver with Master: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857)
I20260927 07:07:23.115696 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:59027
I20260927 07:07:23.120766 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:07:23.127511 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.122068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.21.2.126:40959)
0927 07:07:23.122128 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.122129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.122271 (+ 142us) server_negotiation.cc:407] Connection header received
0927 07:07:23.122289 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.122290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.122299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.122335 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.122804 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.123248 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.124012 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.124182 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.124234 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.125475 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:23.125476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:23.125476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:23.125481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:23.126530 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.126831 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.126831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.126832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.126876 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.127012 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.127012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.127012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.127155 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:23.127223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.127306 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.127417 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:07:23.129099 4451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:23.130779 4451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:23.136885 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.132875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36743 (local address 127.0.0.1:50210)
0927 07:07:23.133169 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.133179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.133251 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.133454 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.133458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.133460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:23.133525 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.133527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.134561 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.134562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.135716 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.135719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136555 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.136633 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.136646 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.136753 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.136809 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":53,"threads_started":1}
I20260927 07:07:23.136898 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.133026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.21.2.65:36743)
0927 07:07:23.133200 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.133205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.133247 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:07:23.133325 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.133328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.133345 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.133425 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:23.133673 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.134456 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.135808 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136456 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136551 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.136612 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.136771 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.136814 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260927 07:07:23.137216 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.133067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:43857 (local address 127.0.0.1:33402)
0927 07:07:23.133241 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.133251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.133291 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.133633 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.133633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.133636 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:23.133678 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.133679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.135084 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.135085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.136060 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.136063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136944 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.137010 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.137014 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.137044 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.137094 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":54,"threads_started":1}
I20260927 07:07:23.137429 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.132954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.21.2.66:46477)
0927 07:07:23.133497 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.133502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.133511 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:07:23.133545 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.133546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.133559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.133644 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:23.133871 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.134938 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136195 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136855 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136916 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.136968 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.137303 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.137352 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":88,"threads_started":1}
I20260927 07:07:23.137238 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.132868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46477 (local address 127.0.0.1:58056)
0927 07:07:23.133042 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.133083 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.133148 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.133753 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.133754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.133755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:23.133788 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.133789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.135052 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.135053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.136130 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.136132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136997 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.137054 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.137057 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.137100 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.137131 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260927 07:07:23.137563 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.133287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.21.2.67:43857)
0927 07:07:23.133469 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.133473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.133482 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:07:23.133517 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.133518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.133534 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.133762 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:23.133837 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.134938 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.136141 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.136837 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.137345 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.137411 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.137436 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.137480 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260927 07:07:23.137972 4577 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0abe5263aa447fb191c1efe40c038b (DEFAULT_TABLE table=test-table [id=05b06a22f33340c6b0dabd6238f5d89e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:23.138353 4577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0abe5263aa447fb191c1efe40c038b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:23.139992 4845 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0abe5263aa447fb191c1efe40c038b (DEFAULT_TABLE table=test-table [id=05b06a22f33340c6b0dabd6238f5d89e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:23.140076 4711 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0abe5263aa447fb191c1efe40c038b (DEFAULT_TABLE table=test-table [id=05b06a22f33340c6b0dabd6238f5d89e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:23.140321 4845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0abe5263aa447fb191c1efe40c038b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:23.140385 4711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0abe5263aa447fb191c1efe40c038b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:23.141513 4925 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap starting.
I20260927 07:07:23.142264 4925 tablet_bootstrap.cc:654] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:23.142848 4925 log.cc:824] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:23.143461 4927 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Bootstrap starting.
I20260927 07:07:23.143695 4925 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: No bootstrap required, opened a new log
I20260927 07:07:23.143767 4925 ts_tablet_manager.cc:1397] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:23.143800 4928 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Bootstrap starting.
I20260927 07:07:23.144021 4927 tablet_bootstrap.cc:654] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:23.144586 4928 tablet_bootstrap.cc:654] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:23.144670 4927 log.cc:824] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:23.145205 4928 log.cc:824] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:23.145447 4927 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: No bootstrap required, opened a new log
I20260927 07:07:23.145522 4927 ts_tablet_manager.cc:1397] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:07:23.145959 4928 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: No bootstrap required, opened a new log
I20260927 07:07:23.146075 4928 ts_tablet_manager.cc:1397] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:23.146013 4925 raft_consensus.cc:348] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.146191 4925 raft_consensus.cc:374] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:23.146234 4925 raft_consensus.cc:729] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c4f47b98e24f6cae85fbaa9a37ab63, State: Initialized, Role: FOLLOWER
I20260927 07:07:23.146353 4925 consensus_queue.cc:260] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.146638 4925 ts_tablet_manager.cc:1428] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:23.146659 4643 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
I20260927 07:07:23.147262 4927 raft_consensus.cc:348] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.147487 4927 raft_consensus.cc:374] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:23.147537 4927 raft_consensus.cc:729] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926cddd481dc4140a6926eef415d1f3b, State: Initialized, Role: FOLLOWER
I20260927 07:07:23.147666 4927 consensus_queue.cc:260] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.148733 4777 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
I20260927 07:07:23.148731 4927 ts_tablet_manager.cc:1428] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:23.149295 4928 raft_consensus.cc:348] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.149482 4928 raft_consensus.cc:374] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:23.149519 4928 raft_consensus.cc:729] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, State: Initialized, Role: FOLLOWER
I20260927 07:07:23.149642 4928 consensus_queue.cc:260] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.149922 4911 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
I20260927 07:07:23.149931 4928 ts_tablet_manager.cc:1428] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260927 07:07:23.204285 4778 tablet.cc:2369] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:23.281975 4931 raft_consensus.cc:482] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:23.282120 4931 raft_consensus.cc:504] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.282454 4931 leader_election.cc:288] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477), 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857)
I20260927 07:07:23.286455 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.282742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:43761 (local address 127.21.2.67:43857)
0927 07:07:23.282795 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.282797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.282897 (+ 100us) server_negotiation.cc:407] Connection header received
0927 07:07:23.282927 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.282928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.282940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.282996 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:23.283136 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.283903 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.285474 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.286091 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.286157 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.286169 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.286340 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.286382 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:07:23.286455 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.282736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:43073 (local address 127.21.2.66:46477)
0927 07:07:23.282795 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.282797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.282807 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:07:23.282844 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.282845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.282853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.282944 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:23.283137 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.283919 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.285473 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.286074 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.286157 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.286169 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.286340 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.286382 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:07:23.286492 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.282627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46477 (local address 127.21.2.65:43073)
0927 07:07:23.282672 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.282685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.282739 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.282964 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.282965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.282967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:23.283057 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.283059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.283997 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.283998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.285354 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.285358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.286172 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.286173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.286257 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.286261 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.286319 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.286372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":104}
I20260927 07:07:23.286499 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.282627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:43857 (local address 127.21.2.65:43761)
0927 07:07:23.282823 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.282839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.282881 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.283004 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.283005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.283006 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:23.283064 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.283065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.283997 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.283998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.285354 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.285358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.286172 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.286173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.286257 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.286261 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:23.286319 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:07:23.286373 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":115,"thread_start_us":94,"threads_started":1}
I20260927 07:07:23.286870 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0abe5263aa447fb191c1efe40c038b" candidate_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cddd481dc4140a6926eef415d1f3b" is_pre_election: true
I20260927 07:07:23.286897 4865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0abe5263aa447fb191c1efe40c038b" candidate_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" is_pre_election: true
I20260927 07:07:23.287037 4731 raft_consensus.cc:2454] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c4f47b98e24f6cae85fbaa9a37ab63 in term 0.
I20260927 07:07:23.287074 4865 raft_consensus.cc:2454] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c4f47b98e24f6cae85fbaa9a37ab63 in term 0.
I20260927 07:07:23.287299 4533 leader_election.cc:302] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b, e2c4f47b98e24f6cae85fbaa9a37ab63; no voters:
I20260927 07:07:23.287465 4931 raft_consensus.cc:2781] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:23.287525 4931 raft_consensus.cc:482] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:23.287554 4931 raft_consensus.cc:3037] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:23.288321 4931 raft_consensus.cc:504] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.288456 4931 leader_election.cc:288] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 1 election: Requested vote from peers 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477), 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857)
I20260927 07:07:23.288630 4865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0abe5263aa447fb191c1efe40c038b" candidate_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
I20260927 07:07:23.288663 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0abe5263aa447fb191c1efe40c038b" candidate_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cddd481dc4140a6926eef415d1f3b"
I20260927 07:07:23.288708 4865 raft_consensus.cc:3037] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:23.288772 4731 raft_consensus.cc:3037] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:23.289485 4865 raft_consensus.cc:2454] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c4f47b98e24f6cae85fbaa9a37ab63 in term 1.
I20260927 07:07:23.289547 4731 raft_consensus.cc:2454] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c4f47b98e24f6cae85fbaa9a37ab63 in term 1.
I20260927 07:07:23.289691 4533 leader_election.cc:302] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, e2c4f47b98e24f6cae85fbaa9a37ab63; no voters:
I20260927 07:07:23.289779 4931 raft_consensus.cc:2781] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:23.289985 4931 raft_consensus.cc:686] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 LEADER]: Becoming Leader. State: Replica: e2c4f47b98e24f6cae85fbaa9a37ab63, State: Running, Role: LEADER
I20260927 07:07:23.290091 4931 consensus_queue.cc:237] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:23.290948 4451 catalog_manager.cc:5495] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } health_report { overall_health: UNKNOWN } } }
W20260927 07:07:23.304601 4644 tablet.cc:2369] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:23.318041 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.315885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.21.2.65:36743)
0927 07:07:23.315948 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.315950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.315957 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:07:23.316004 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.316005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.316024 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.316077 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:23.316183 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.316637 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.317351 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.317531 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.317580 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.317715 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:23.317810 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:23.317869 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.317907 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.317923 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:07:23.319851 4865 raft_consensus.cc:1261] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Refusing update from remote peer e2c4f47b98e24f6cae85fbaa9a37ab63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:23.319876 4731 raft_consensus.cc:1261] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Refusing update from remote peer e2c4f47b98e24f6cae85fbaa9a37ab63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:23.320241 4931 consensus_queue.cc:1035] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:23.320401 4935 consensus_queue.cc:1035] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:23.322687 4938 mvcc.cc:205] Tried to move back new op lower bound from 7333858686235238400 to 7333858686116966400. Current Snapshot: MvccSnapshot[applied={T|T < 7333858686235238400}]
I20260927 07:07:23.324993 4451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:23.325098 4451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:23.325841 4451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Sending DeleteTablet for 3 replicas of tablet 4c0abe5263aa447fb191c1efe40c038b
I20260927 07:07:23.326148 4711 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0abe5263aa447fb191c1efe40c038b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:23 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:58056
I20260927 07:07:23.326169 4845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0abe5263aa447fb191c1efe40c038b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:23 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:33402
I20260927 07:07:23.326432 4948 tablet_replica.cc:330] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: stopping tablet replica
I20260927 07:07:23.326422 4577 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0abe5263aa447fb191c1efe40c038b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:23 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:50210
I20260927 07:07:23.326552 4948 raft_consensus.cc:2229] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:23.326671 4950 tablet_replica.cc:330] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: stopping tablet replica
I20260927 07:07:23.326668 4948 raft_consensus.cc:2258] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:23.326784 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4514
I20260927 07:07:23.326813 4950 raft_consensus.cc:2229] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:23.326997 4950 raft_consensus.cc:2258] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:23.327553 4949 tablet_replica.cc:330] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: stopping tablet replica
I20260927 07:07:23.327669 4949 raft_consensus.cc:2229] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:23.327795 4949 raft_consensus.cc:2258] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:23.328182 4949 ts_tablet_manager.cc:1905] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:23.328514 4948 ts_tablet_manager.cc:1905] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:23.329766 4949 ts_tablet_manager.cc:1918] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:23.329851 4949 log.cc:1196] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/4c0abe5263aa447fb191c1efe40c038b
I20260927 07:07:23.330133 4949 ts_tablet_manager.cc:1939] T 4c0abe5263aa447fb191c1efe40c038b P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting consensus metadata
I20260927 07:07:23.330428 4948 ts_tablet_manager.cc:1918] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:23.330502 4948 log.cc:1196] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/4c0abe5263aa447fb191c1efe40c038b
I20260927 07:07:23.330556 4439 catalog_manager.cc:4841] TS 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): tablet 4c0abe5263aa447fb191c1efe40c038b (table test-table [id=05b06a22f33340c6b0dabd6238f5d89e]) successfully deleted
I20260927 07:07:23.330753 4948 ts_tablet_manager.cc:1939] T 4c0abe5263aa447fb191c1efe40c038b P 926cddd481dc4140a6926eef415d1f3b: Deleting consensus metadata
I20260927 07:07:23.331156 4437 catalog_manager.cc:4841] TS 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): tablet 4c0abe5263aa447fb191c1efe40c038b (table test-table [id=05b06a22f33340c6b0dabd6238f5d89e]) successfully deleted
W20260927 07:07:23.333794 4439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:36743 (error 108)
I20260927 07:07:23.334026 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4648
I20260927 07:07:23.334841 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.334184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36743 (local address 127.0.0.1:50224)
0927 07:07:23.334773 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.334811 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569}
W20260927 07:07:23.335002 4439 catalog_manager.cc:4584] TS e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c0abe5263aa447fb191c1efe40c038b: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36743: connect: Connection refused (error 111)
I20260927 07:07:23.339247 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4782
I20260927 07:07:23.343807 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4420
I20260927 07:07:43.349851 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--webserver_interface=127.21.2.126
--webserver_port=38237
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:43.430830 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:43.431030 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:43.431051 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:43.432126 4952 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:07:43.432160 4952 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:07:43.432173 4952 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:07:43.432185 4952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:43.432197 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:43.432210 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:43.432222 4952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:43.432233 4952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:43.433604 4952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=38237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:43.433789 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:43.433964 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:43.442999 4961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.442976 4958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.443022 4952 server_base.cc:1034] running on GCE node
W20260927 07:07:43.442953 4959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.443557 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:43.443884 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:43.445029 4952 hybrid_clock.cc:648] HybridClock initialized: now 1790492863445009 us; error 38 us; skew 500 ppm
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:43.452651 4952 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:07:43.453145 4952 webserver.cc:460] Webserver started at http://127.21.2.126:38237/ using document root <none> and password file <none>
I20260927 07:07:43.453400 4952 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:43.453474 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:43.455259 4952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:43.455950 4968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.456146 4952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.456241 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "441dc8a7f29e4ae181d594a0318550d6"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "d2ebc4f42b361cfb8f43e8b2ca5f526a"
server_key_iv: "a281e4e37b0228f044af9ca3e057d476"
server_key_version: "encryptionkey@0"
I20260927 07:07:43.456642 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:43.471568 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:43.473400 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:43.473549 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:43.478068 4952 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40959
I20260927 07:07:43.478127 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40959 every 8 connection(s)
I20260927 07:07:43.478487 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:43.479982 5021 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:43.480633 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap starting.
I20260927 07:07:43.485153 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 4952
I20260927 07:07:43.485481 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:36743
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=43433
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:43.489428 5021 log.cc:824] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:43.491246 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:43.491590 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap complete.
I20260927 07:07:43.493815 5021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:43.494063 5021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Initialized, Role: FOLLOWER
I20260927 07:07:43.494266 5021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:43.494357 5021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:43.494413 5021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:43.494459 5021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:07:43.495502 5021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:43.495613 5021 leader_election.cc:302] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441dc8a7f29e4ae181d594a0318550d6; no voters:
I20260927 07:07:43.495806 5021 leader_election.cc:288] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:07:43.495842 5025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:07:43.495961 5025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 LEADER]: Becoming Leader. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Running, Role: LEADER
I20260927 07:07:43.496078 5021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:43.496048 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:07:43.496343 5026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:07:43.496431 5026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:43.496652 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441dc8a7f29e4ae181d594a0318550d6. Latest consensus state: current_term: 2 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:07:43.496749 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:43.496883 5030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:43.497618 5030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:43.497716 5030 catalog_manager.cc:1259] Loaded cluster ID: edd8cc8326114f4c8554b297155ab5fa
I20260927 07:07:43.497993 5030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:43.499577 5030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:43.499755 5030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Loaded TSK: 0
I20260927 07:07:43.509786 5030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Generated new TSK 1
I20260927 07:07:43.509884 5030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:07:43.574399 5023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:43.574609 5023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:43.574633 5023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:43.575870 5023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:43.575914 5023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:43.575941 5023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:07:43.577378 5023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:36743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=43433
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:43.577638 5023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:43.577874 5023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:43.582837 5048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.582840 5049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.582857 5051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.583484 5023 server_base.cc:1034] running on GCE node
I20260927 07:07:43.583712 5023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:43.583952 5023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:43.585151 5023 hybrid_clock.cc:648] HybridClock initialized: now 1790492863585136 us; error 32 us; skew 500 ppm
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:43.587814 5023 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:07:43.588327 5023 webserver.cc:460] Webserver started at http://127.21.2.65:43433/ using document root <none> and password file <none>
I20260927 07:07:43.588574 5023 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:43.588658 5023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:43.590034 5023 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.590746 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.590898 5023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.591001 5023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "e8b8d64567c9f98cc3a5969d8c355db0"
server_key_iv: "3ba08a02ca74492e37c35886403693bc"
server_key_version: "encryptionkey@0"
I20260927 07:07:43.591398 5023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:43.600240 5023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:43.602448 5023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:43.602592 5023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:43.602819 5023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:43.603281 5065 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:07:43.605223 5023 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:07:43.605283 5023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:07:43.605326 5023 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:07:43.605918 5023 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:07:43.605975 5023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:07:43.606046 5065 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap starting.
I20260927 07:07:43.611346 5023 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:36743
I20260927 07:07:43.611908 5023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:43.616920 5065 log.cc:824] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:43.616935 5172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:36743 every 8 connection(s)
I20260927 07:07:43.619910 5065 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:43.620222 5065 tablet_bootstrap.cc:492] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap complete.
I20260927 07:07:43.621094 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5023
I20260927 07:07:43.621349 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:46477
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=40149
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:43.621364 5065 ts_tablet_manager.cc:1397] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260927 07:07:43.623142 5065 raft_consensus.cc:348] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:43.623495 5065 raft_consensus.cc:729] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c4f47b98e24f6cae85fbaa9a37ab63, State: Initialized, Role: FOLLOWER
I20260927 07:07:43.623677 5065 consensus_queue.cc:260] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } } peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } }
I20260927 07:07:43.623978 5065 ts_tablet_manager.cc:1428] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:43.631563 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.617645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:50817)
0927 07:07:43.617802 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.617855 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.618483 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.618963 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.618968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.619339 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:43.619637 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.619643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.621390 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.621392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.623643 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.623647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.624442 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.624444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.624595 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.625274 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:43.625284 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:43.627213 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:43.629691 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.629698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.629700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.629985 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.630291 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.630292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.630292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.630414 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.630896 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:43.630901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:43.631043 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.631148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.631305 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260927 07:07:43.631984 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.617732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:50817 (local address 127.21.2.126:40959)
0927 07:07:43.617927 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.617936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.617978 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:07:43.618716 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.618724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.618778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.618912 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:43.619764 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.621260 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.623748 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.624340 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.624445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.627335 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:43.627340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:43.627340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:43.627354 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:43.629587 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.630083 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.630087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.630087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.630213 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.630487 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.630489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.630489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.630818 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:43.630916 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.631230 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.631417 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260927 07:07:43.632648 5173 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:43.632757 5173 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:43.632953 5173 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:43.633769 4985 ts_manager.cc:194] Registered new tserver with Master: e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743)
I20260927 07:07:43.635025 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:50817
I20260927 07:07:43.636044 5173 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
I20260927 07:07:43.638952 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.634442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36743 (local address 127.0.0.1:57158)
0927 07:07:43.635669 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.635699 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.635759 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.636034 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.636036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.636039 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:43.636105 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.636107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.637148 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.637149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.637930 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.637932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.638626 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.638628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.638719 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.638730 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.638819 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.638877 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"thread_start_us":95,"threads_started":1}
I20260927 07:07:43.638962 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.634697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.21.2.65:36743)
0927 07:07:43.635872 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.635876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.635883 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:07:43.635908 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.635909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.635920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.636015 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:43.636250 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.637058 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.638024 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.638534 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.638624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.638681 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.638838 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.638877 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":73,"threads_started":1}
I20260927 07:07:43.639611 5107 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0abe5263aa447fb191c1efe40c038b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:23 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:57158
I20260927 07:07:43.640292 5182 tablet_replica.cc:330] stopping tablet replica
I20260927 07:07:43.640417 5182 raft_consensus.cc:2229] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:43.640491 5182 raft_consensus.cc:2258] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:43.640811 5182 ts_tablet_manager.cc:1905] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:43.643270 5182 ts_tablet_manager.cc:1918] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:43.643337 5182 log.cc:1196] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal/wals/4c0abe5263aa447fb191c1efe40c038b
I20260927 07:07:43.643620 5182 ts_tablet_manager.cc:1939] T 4c0abe5263aa447fb191c1efe40c038b P e2c4f47b98e24f6cae85fbaa9a37ab63: Deleting consensus metadata
I20260927 07:07:43.644143 4972 catalog_manager.cc:4841] TS e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743): tablet 4c0abe5263aa447fb191c1efe40c038b (table test-table [id=05b06a22f33340c6b0dabd6238f5d89e]) successfully deleted
W20260927 07:07:43.708348 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:43.708588 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:43.708624 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:43.709769 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:43.709836 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:43.709882 5178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:07:43.711336 5178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:46477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=40149
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:43.711624 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:43.711851 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:43.716742 5187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.716751 5190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.716751 5188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.717144 5178 server_base.cc:1034] running on GCE node
I20260927 07:07:43.717360 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:43.717602 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:43.718782 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790492863718770 us; error 28 us; skew 500 ppm
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:43.721305 5178 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:07:43.721686 5178 webserver.cc:460] Webserver started at http://127.21.2.66:40149/ using document root <none> and password file <none>
I20260927 07:07:43.721863 5178 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:43.721913 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:43.723210 5178 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.723800 5197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.723968 5178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.724056 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "926cddd481dc4140a6926eef415d1f3b"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "fdce70011149f28a8ae4bf7b4333282d"
server_key_iv: "ca56076e33d1e0c5140d70d8c717cb6b"
server_key_version: "encryptionkey@0"
I20260927 07:07:43.724388 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:43.733063 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:43.735285 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:43.735476 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:43.735735 5178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:43.736052 5178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:43.736119 5178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.736161 5178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:43.736189 5178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.741870 5178 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46477
I20260927 07:07:43.741936 5310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46477 every 8 connection(s)
I20260927 07:07:43.742291 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:43.748135 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5178
I20260927 07:07:43.748400 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:43857
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=33607
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:43.751920 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.743074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:39597 (local address 127.21.2.126:40959)
0927 07:07:43.743141 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.743143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.743359 (+ 216us) server_negotiation.cc:407] Connection header received
0927 07:07:43.743932 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.743933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.743949 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.743992 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:43.744941 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.745440 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.746458 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.746707 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.746798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.748996 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:43.748998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:43.748998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:43.749007 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:43.750502 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.750894 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.750895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.750896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.750961 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.751211 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.751212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.751213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.751345 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:43.751475 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.751738 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.751828 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:07:43.752130 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.743039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:39597)
0927 07:07:43.743237 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.743278 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.743817 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.744065 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.744070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.744647 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:43.744829 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.744834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.745548 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.745549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.746346 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.746349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.746811 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.746812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.746939 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.747455 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:43.747463 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:43.748905 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:43.750592 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.750599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.750601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.750817 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.751032 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.751033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.751033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.751142 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.751456 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:43.751464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:43.751609 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.751717 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.751870 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260927 07:07:43.752579 5311 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:43.752672 5311 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:43.752866 5311 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:43.753235 4985 ts_manager.cc:194] Registered new tserver with Master: 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
I20260927 07:07:43.753613 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:39597
W20260927 07:07:43.830873 5314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:43.831115 5314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:43.831154 5314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:43.832322 5314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:43.832388 5314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:43.832434 5314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:07:43.833962 5314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=33607
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:07:43.834232 5314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:43.834510 5314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:43.839283 5320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.839313 5314 server_base.cc:1034] running on GCE node
W20260927 07:07:43.839455 5319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:43.839313 5322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:43.839713 5314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:43.839972 5314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:43.841145 5314 hybrid_clock.cc:648] HybridClock initialized: now 1790492863841149 us; error 43 us; skew 500 ppm
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:43.843787 5314 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:07:43.844288 5314 webserver.cc:460] Webserver started at http://127.21.2.67:33607/ using document root <none> and password file <none>
I20260927 07:07:43.844530 5314 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:43.844619 5314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:43.846060 5314 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.846735 5329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.846909 5314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.847028 5314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
format_stamp: "Formatted at 2026-09-27 07:07:23 on dist-test-slave-ltqz"
server_key: "47e1171263d9cd15390509f9bf783b9f"
server_key_iv: "d8c4246491633530a5763eb795e057a8"
server_key_version: "encryptionkey@0"
I20260927 07:07:43.847402 5314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:43.865852 5314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:43.867980 5314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:43.868157 5314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:43.868394 5314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:43.868717 5314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:43.868778 5314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.868821 5314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:43.868848 5314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.874727 5314 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:43857
I20260927 07:07:43.874779 5442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:43857 every 8 connection(s)
I20260927 07:07:43.875125 5314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:07:43 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492863, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:07:43.883855 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5314
I20260927 07:07:43.884123 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.875865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:49085 (local address 127.21.2.126:40959)
0927 07:07:43.875933 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.875935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.876146 (+ 211us) server_negotiation.cc:407] Connection header received
0927 07:07:43.876651 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.876652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.876661 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.876718 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:43.877587 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.878058 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.878995 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.879173 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.879224 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.881316 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:43.881318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:43.881318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:43.881325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:43.882734 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.883163 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.883164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.883165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.883216 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.883513 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.883514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.883515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.883626 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:43.883697 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.883934 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.884022 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:07:43.884366 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.875850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:49085)
0927 07:07:43.876021 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.876062 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.876563 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.876767 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.876770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.877324 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:43.877512 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.877516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.878142 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.878143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.878919 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.878921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.879244 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.879244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.879339 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.879844 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:43.879852 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:43.881223 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:43.882828 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.882834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.882836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.883082 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.883278 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:43.883279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:43.883279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:43.883447 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:43.883680 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:43.883689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:43.883802 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.883884 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.884106 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260927 07:07:43.884958 5443 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40959
I20260927 07:07:43.885075 5443 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:43.885259 5443 heartbeater.cc:511] Master 127.21.2.126:40959 requested a full tablet report, sending...
I20260927 07:07:43.885651 4985 ts_manager.cc:194] Registered new tserver with Master: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857)
I20260927 07:07:43.886019 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:49085
I20260927 07:07:43.890434 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.884499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.21.2.126:40959)
0927 07:07:43.884539 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.884541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.884552 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:07:43.884615 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.884616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.884627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.884684 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:43.885385 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.886009 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.886864 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.887120 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.887179 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.888423 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:43.888424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:43.888424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:43.888430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:43.889481 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.889781 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.889782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.889782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.889828 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:43.889985 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:43.889985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:43.889985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:43.890073 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:43.890201 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.890311 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.890366 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:07:43.891137 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:07:43.893553 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.891505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.21.2.126:40959)
0927 07:07:43.891556 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.891557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.891638 (+ 81us) server_negotiation.cc:407] Connection header received
0927 07:07:43.891683 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.891684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.891709 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.891771 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:43.891843 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.892286 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.892970 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.893136 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.893207 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.893314 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:43.893402 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:43.893464 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.893494 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.893508 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 07:07:43.893859 4985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:43.895620 4985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:43.897740 5107 tablet_service.cc:1467] Processing CreateTablet for tablet e1e9e459488842a495a2fa479cfb101a (DEFAULT_TABLE table=test-table [id=3ea6a5e06a624ea4956be178a4760e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:43.897945 5107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e9e459488842a495a2fa479cfb101a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:43.899659 5065 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap starting.
I20260927 07:07:43.900082 5065 tablet_bootstrap.cc:654] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:43.900764 5065 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: No bootstrap required, opened a new log
I20260927 07:07:43.900820 5065 ts_tablet_manager.cc:1397] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:07:43.900960 5065 raft_consensus.cc:348] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.901046 5065 raft_consensus.cc:374] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:43.901096 5065 raft_consensus.cc:729] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c4f47b98e24f6cae85fbaa9a37ab63, State: Initialized, Role: FOLLOWER
I20260927 07:07:43.901142 5065 consensus_queue.cc:260] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.901257 5065 ts_tablet_manager.cc:1428] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:43.901417 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.897396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:43857 (local address 127.0.0.1:33190)
0927 07:07:43.897450 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.897461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.897497 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.898053 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.898055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.898057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:43.898096 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.898097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.899109 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.899110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.900116 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.900119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.901174 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.901175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.901235 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.901239 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.901297 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.901340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 07:07:43.901508 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.897593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.21.2.67:43857)
0927 07:07:43.897777 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.897782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.897793 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:07:43.897926 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.897928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.897944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.898186 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:43.898270 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.899027 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.900281 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.900983 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.901059 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.901122 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.901378 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.901424 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260927 07:07:43.901786 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.897434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46477 (local address 127.0.0.1:55956)
0927 07:07:43.897765 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:43.897776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:43.897824 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:43.898119 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:43.898120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:43.898122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:43.898156 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.898157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.899489 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.899490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.900589 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:43.900592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.901554 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.901555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:43.901631 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.901636 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:07:43.901688 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:07:43.901723 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":58,"threads_started":1}
I20260927 07:07:43.902070 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.897621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.21.2.66:46477)
0927 07:07:43.897860 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.897864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.897872 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:43.897908 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.897910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.897927 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.898029 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:43.898240 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.899390 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.900668 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:43.901384 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:43.901457 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:43.901516 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.901946 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.901990 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":132,"threads_started":1}
I20260927 07:07:43.902266 5377 tablet_service.cc:1467] Processing CreateTablet for tablet e1e9e459488842a495a2fa479cfb101a (DEFAULT_TABLE table=test-table [id=3ea6a5e06a624ea4956be178a4760e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:43.902606 5377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e9e459488842a495a2fa479cfb101a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:43.902757 5245 tablet_service.cc:1467] Processing CreateTablet for tablet e1e9e459488842a495a2fa479cfb101a (DEFAULT_TABLE table=test-table [id=3ea6a5e06a624ea4956be178a4760e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:43.903064 5245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e9e459488842a495a2fa479cfb101a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:43.906239 5452 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Bootstrap starting.
I20260927 07:07:43.906438 5453 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Bootstrap starting.
I20260927 07:07:43.907101 5453 tablet_bootstrap.cc:654] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:43.907101 5452 tablet_bootstrap.cc:654] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:43.907755 5452 log.cc:824] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:43.907795 5453 log.cc:824] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:43.908571 5452 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: No bootstrap required, opened a new log
I20260927 07:07:43.908643 5452 ts_tablet_manager.cc:1397] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:07:43.908713 5453 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: No bootstrap required, opened a new log
I20260927 07:07:43.908779 5453 ts_tablet_manager.cc:1397] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:07:43.910646 5452 raft_consensus.cc:348] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.910646 5453 raft_consensus.cc:348] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.910836 5453 raft_consensus.cc:374] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:43.910836 5452 raft_consensus.cc:374] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:43.910876 5452 raft_consensus.cc:729] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, State: Initialized, Role: FOLLOWER
I20260927 07:07:43.910876 5453 raft_consensus.cc:729] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926cddd481dc4140a6926eef415d1f3b, State: Initialized, Role: FOLLOWER
I20260927 07:07:43.911003 5453 consensus_queue.cc:260] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.911003 5452 consensus_queue.cc:260] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:43.911298 5453 ts_tablet_manager.cc:1428] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 07:07:43.911298 5452 ts_tablet_manager.cc:1428] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:07:43.911306 5443 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
I20260927 07:07:43.911412 5311 heartbeater.cc:503] Master 127.21.2.126:40959 was elected leader, sending a full tablet report...
W20260927 07:07:43.992887 5312 tablet.cc:2369] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:44.025693 5457 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:44.025830 5457 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:44.026294 5457 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743)
I20260927 07:07:44.029456 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.026563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:56491 (local address 127.21.2.67:43857)
0927 07:07:44.026621 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:44.026636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:44.026666 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:07:44.026696 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:44.026697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:44.026731 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:44.026779 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:44.027008 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.027631 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.028693 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.029073 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.029130 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.029138 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:44.029336 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 07:07:44.029364 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:07:44.029467 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.026459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:43857 (local address 127.21.2.66:56491)
0927 07:07:44.026531 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:44.026545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:44.026614 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:44.026804 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:44.026807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:44.026814 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:44.026899 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:44.026901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.027708 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.027709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:44.028597 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:44.028599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.029151 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.029152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:44.029248 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.029253 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:07:44.029325 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:07:44.029373 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":60}
I20260927 07:07:44.029829 5397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e9e459488842a495a2fa479cfb101a" candidate_uuid: "926cddd481dc4140a6926eef415d1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" is_pre_election: true
I20260927 07:07:44.030016 5397 raft_consensus.cc:2454] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926cddd481dc4140a6926eef415d1f3b in term 0.
I20260927 07:07:44.030119 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.026598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36743 (local address 127.21.2.66:53191)
0927 07:07:44.026800 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:44.026810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:44.026866 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:44.027019 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:44.027020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:44.027022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:44.027088 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:44.027090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.027763 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.027764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:44.028600 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:44.028602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.029841 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.029842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:44.029907 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.029910 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:07:44.029995 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 07:07:44.030032 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260927 07:07:44.030143 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.026704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:53191 (local address 127.21.2.65:36743)
0927 07:07:44.026755 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:44.026757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:44.026904 (+ 147us) server_negotiation.cc:407] Connection header received
0927 07:07:44.026933 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:44.026934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:44.026948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:44.026999 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:44.027160 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.027664 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.028710 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.029756 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.029823 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.029833 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:44.030016 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 07:07:44.030069 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:07:44.030272 5201 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b; no voters:
I20260927 07:07:44.030404 5457 raft_consensus.cc:2781] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:44.030450 5457 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:44.030583 5127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e9e459488842a495a2fa479cfb101a" candidate_uuid: "926cddd481dc4140a6926eef415d1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" is_pre_election: true
I20260927 07:07:44.030735 5127 raft_consensus.cc:2454] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926cddd481dc4140a6926eef415d1f3b in term 0.
I20260927 07:07:44.030467 5457 raft_consensus.cc:3037] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:44.031657 5457 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:44.031783 5457 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [CANDIDATE]: Term 1 election: Requested vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743)
I20260927 07:07:44.031973 5397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e9e459488842a495a2fa479cfb101a" candidate_uuid: "926cddd481dc4140a6926eef415d1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
I20260927 07:07:44.032039 5127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e9e459488842a495a2fa479cfb101a" candidate_uuid: "926cddd481dc4140a6926eef415d1f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
I20260927 07:07:44.032076 5397 raft_consensus.cc:3037] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:44.032109 5127 raft_consensus.cc:3037] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:44.032927 5127 raft_consensus.cc:2454] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926cddd481dc4140a6926eef415d1f3b in term 1.
I20260927 07:07:44.032946 5397 raft_consensus.cc:2454] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926cddd481dc4140a6926eef415d1f3b in term 1.
I20260927 07:07:44.033135 5201 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926cddd481dc4140a6926eef415d1f3b, e2c4f47b98e24f6cae85fbaa9a37ab63; no voters:
I20260927 07:07:44.033226 5457 raft_consensus.cc:2781] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:44.033367 5457 raft_consensus.cc:686] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 1 LEADER]: Becoming Leader. State: Replica: 926cddd481dc4140a6926eef415d1f3b, State: Running, Role: LEADER
I20260927 07:07:44.033457 5457 consensus_queue.cc:237] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:07:44.034317 4984 catalog_manager.cc:5495] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 926cddd481dc4140a6926eef415d1f3b (127.21.2.66). New cstate: current_term: 1 leader_uuid: "926cddd481dc4140a6926eef415d1f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:44.050482 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.047778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.21.2.66:46477)
0927 07:07:44.047838 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:44.047840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:44.047850 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:07:44.047972 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:44.047973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:44.047998 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:44.048090 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:44.048227 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.048705 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.049507 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.049719 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.049793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.049978 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:44.050037 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 07:07:44.050349 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55972: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 07:07:44.050573 5450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55972: token expired
I20260927 07:07:44.057768 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.051139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.21.2.126:40959)
0927 07:07:44.051182 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:44.051184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:44.051192 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:44.051214 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:44.051215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:44.051223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:44.051271 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:44.051821 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.052301 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.053160 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.053513 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.053590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.054944 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:44.054946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:44.054946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:44.054955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:44.056522 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:44.056898 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:44.056900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:44.056900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:44.056966 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:44.057179 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:44.057180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:44.057181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:44.057343 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:44.057469 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:44.057570 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:07:44.057659 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:07:44.061184 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.059045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.21.2.66:46477)
0927 07:07:44.059105 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:44.059107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:44.059115 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:07:44.059139 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:44.059140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:44.059153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:44.059235 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:44.059297 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.059881 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.060583 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:44.060755 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:44.060806 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:44.060936 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:44.061029 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:44.061069 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:44.061113 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 07:07:44.061131 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:07:44.063057 5127 raft_consensus.cc:1261] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Refusing update from remote peer 926cddd481dc4140a6926eef415d1f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:44.063074 5397 raft_consensus.cc:1261] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Refusing update from remote peer 926cddd481dc4140a6926eef415d1f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:44.063496 5457 consensus_queue.cc:1035] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:44.063632 5459 consensus_queue.cc:1035] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:44.065757 5462 mvcc.cc:205] Tried to move back new op lower bound from 7333858771199422464 to 7333858771081670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333858771199422464}]
I20260927 07:07:44.065900 5463 mvcc.cc:205] Tried to move back new op lower bound from 7333858771199422464 to 7333858771081670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333858771199422464}]
I20260927 07:07:44.067270 5464 mvcc.cc:205] Tried to move back new op lower bound from 7333858771199422464 to 7333858771081670656. Current Snapshot: MvccSnapshot[applied={T|T < 7333858771199422464}]
I20260927 07:07:44.068470 4984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:44.068568 4984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:44.069358 4984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Sending DeleteTablet for 3 replicas of tablet e1e9e459488842a495a2fa479cfb101a
I20260927 07:07:44.069713 5377 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e9e459488842a495a2fa479cfb101a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:44 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:33190
I20260927 07:07:44.069715 5245 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e9e459488842a495a2fa479cfb101a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:44 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:55956
I20260927 07:07:44.069736 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5023
I20260927 07:07:44.071586 5472 tablet_replica.cc:330] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: stopping tablet replica
I20260927 07:07:44.071704 5472 raft_consensus.cc:2229] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:44.071813 5472 raft_consensus.cc:2258] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:44.072294 5472 ts_tablet_manager.cc:1905] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:44.074270 5472 ts_tablet_manager.cc:1918] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:44.074349 5472 log.cc:1196] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/e1e9e459488842a495a2fa479cfb101a
I20260927 07:07:44.074671 5472 ts_tablet_manager.cc:1939] T e1e9e459488842a495a2fa479cfb101a P 926cddd481dc4140a6926eef415d1f3b: Deleting consensus metadata
I20260927 07:07:44.075112 4970 catalog_manager.cc:4841] TS 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): tablet e1e9e459488842a495a2fa479cfb101a (table test-table [id=3ea6a5e06a624ea4956be178a4760e9d]) successfully deleted
W20260927 07:07:44.075775 4972 connection.cc:535] client connection to 127.21.2.65:36743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 07:07:44.075886 4972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:07:44.076438 5473 tablet_replica.cc:330] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: stopping tablet replica
I20260927 07:07:44.076524 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:44.076363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:36743 (local address 127.0.0.1:57162)
0927 07:07:44.076439 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:44.076491 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 07:07:44.076570 5473 raft_consensus.cc:2229] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:44.076687 5473 raft_consensus.cc:2258] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:07:44.076719 4972 catalog_manager.cc:4584] TS e2c4f47b98e24f6cae85fbaa9a37ab63 (127.21.2.65:36743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e9e459488842a495a2fa479cfb101a: Network error: Client connection negotiation failed: client connection to 127.21.2.65:36743: connect: Connection refused (error 111)
I20260927 07:07:44.077173 5473 ts_tablet_manager.cc:1905] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:44.077289 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5178
I20260927 07:07:44.078890 5473 ts_tablet_manager.cc:1918] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:44.078944 5473 log.cc:1196] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/e1e9e459488842a495a2fa479cfb101a
I20260927 07:07:44.079155 5473 ts_tablet_manager.cc:1939] T e1e9e459488842a495a2fa479cfb101a P 77ff79a2cc8440d8ad0fc0a2ec6fc7f2: Deleting consensus metadata
I20260927 07:07:44.079691 4972 catalog_manager.cc:4841] TS 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): tablet e1e9e459488842a495a2fa479cfb101a (table test-table [id=3ea6a5e06a624ea4956be178a4760e9d]) successfully deleted
I20260927 07:07:44.083396 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5314
I20260927 07:07:44.088145 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 4952
I20260927 07:08:04.095667 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--webserver_interface=127.21.2.126
--webserver_port=38237
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:04.181666 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:04.181877 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:04.181901 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:04.182996 5475 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:08:04.183029 5475 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:08:04.183043 5475 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:08:04.183055 5475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:04.183068 5475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:04.183081 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:04.183094 5475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:04.183106 5475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:04.184544 5475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=38237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:04.184747 5475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:04.184921 5475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:04.190155 5475 server_base.cc:1034] running on GCE node
W20260927 07:08:04.190124 5483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.190124 5480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.190165 5481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:04.190570 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:04.190853 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:04.192013 5475 hybrid_clock.cc:648] HybridClock initialized: now 1790492884191999 us; error 31 us; skew 500 ppm
Sep 27 07:08:04 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:04.194789 5475 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:04.195257 5475 webserver.cc:460] Webserver started at http://127.21.2.126:38237/ using document root <none> and password file <none>
I20260927 07:08:04.195518 5475 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:04.195590 5475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:04.196878 5475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:04.197546 5490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.197715 5475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:04.197814 5475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "441dc8a7f29e4ae181d594a0318550d6"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "d2ebc4f42b361cfb8f43e8b2ca5f526a"
server_key_iv: "a281e4e37b0228f044af9ca3e057d476"
server_key_version: "encryptionkey@0"
I20260927 07:08:04.198185 5475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:04.209643 5475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:04.211584 5475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:04.211732 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:04.216077 5475 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40959
I20260927 07:08:04.216140 5542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40959 every 8 connection(s)
I20260927 07:08:04.216590 5475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:04.218645 5543 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:08:04.219281 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap starting.
I20260927 07:08:04.220763 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5475
I20260927 07:08:04.220988 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:36743
--local_ip_for_outbound_sockets=127.21.2.65
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=43433
--webserver_interface=127.21.2.65
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:04.228334 5543 log.cc:824] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:04.230609 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:08:04.230940 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: Bootstrap complete.
I20260927 07:08:04.233223 5543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:08:04.233479 5543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Initialized, Role: FOLLOWER
I20260927 07:08:04.233659 5543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:08:04.233784 5543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:04.233846 5543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:04.233901 5543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:08:04.234887 5543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:08:04.235018 5543 leader_election.cc:302] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441dc8a7f29e4ae181d594a0318550d6; no voters:
I20260927 07:08:04.235206 5543 leader_election.cc:288] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:08:04.235280 5547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:08:04.235503 5543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:04.235522 5547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [term 3 LEADER]: Becoming Leader. State: Replica: 441dc8a7f29e4ae181d594a0318550d6, State: Running, Role: LEADER
I20260927 07:08:04.235625 5547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } }
I20260927 07:08:04.235917 5547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441dc8a7f29e4ae181d594a0318550d6. Latest consensus state: current_term: 3 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:08:04.235984 5547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:04.236121 5548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "441dc8a7f29e4ae181d594a0318550d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441dc8a7f29e4ae181d594a0318550d6" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40959 } } }
I20260927 07:08:04.236179 5548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:04.237519 5562 catalog_manager.cc:1259] Loaded cluster ID: edd8cc8326114f4c8554b297155ab5fa
I20260927 07:08:04.237571 5562 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: loading cluster ID for follower catalog manager: success
I20260927 07:08:04.238729 5562 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 441dc8a7f29e4ae181d594a0318550d6: acquiring CA information for follower catalog manager: success
F20260927 07:08:04.239111 5562 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd6a3cfb9ad google::LogMessage::Fail() at ??:0
@ 0x7fd6a3cfe88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd6a3cfb4de google::LogMessage::Flush() at ??:0
@ 0x7fd6a3cfca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd6a6468104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd6abc546b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fd6abc54989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fd6abc54e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fd6abc578c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fd6a5bbe954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd6a6a226db start_thread at ??:0
@ 0x7fd6a27f961f clone at ??:0
W20260927 07:08:04.321512 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:04.321722 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:04.321754 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:04.323021 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:04.323065 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:04.323092 5545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:04.324587 5545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:36743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=43433
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:04.324836 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:04.325081 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:04.330171 5584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.330192 5582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.330189 5581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:04.330524 5545 server_base.cc:1034] running on GCE node
I20260927 07:08:04.330734 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:04.330982 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:04.332177 5545 hybrid_clock.cc:648] HybridClock initialized: now 1790492884332155 us; error 36 us; skew 500 ppm
Sep 27 07:08:04 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:04.334941 5545 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:08:04.335503 5545 webserver.cc:460] Webserver started at http://127.21.2.65:43433/ using document root <none> and password file <none>
I20260927 07:08:04.335745 5545 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:04.335815 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:04.337249 5545 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:04.338008 5591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.338195 5545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:04.338295 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "e8b8d64567c9f98cc3a5969d8c355db0"
server_key_iv: "3ba08a02ca74492e37c35886403693bc"
server_key_version: "encryptionkey@0"
I20260927 07:08:04.338656 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:04.374554 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:04.376959 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:04.377134 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:04.377383 5545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:04.377835 5598 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:08:04.379884 5545 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:08:04.379976 5545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:08:04.380025 5545 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:08:04.380683 5545 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:08:04.380740 5545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:08:04.380817 5598 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap starting.
I20260927 07:08:04.388052 5545 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:36743
I20260927 07:08:04.388234 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:36743 every 8 connection(s)
I20260927 07:08:04.388499 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:04.392459 5598 log.cc:824] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:04.395464 5598 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:08:04.395825 5598 tablet_bootstrap.cc:492] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Bootstrap complete.
I20260927 07:08:04.397010 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5545
I20260927 07:08:04.397058 5598 ts_tablet_manager.cc:1397] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260927 07:08:04.397311 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:46477
--local_ip_for_outbound_sockets=127.21.2.66
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=40149
--webserver_interface=127.21.2.66
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:04.398515 5598 raft_consensus.cc:348] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:04.398871 5598 raft_consensus.cc:729] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c4f47b98e24f6cae85fbaa9a37ab63, State: Initialized, Role: FOLLOWER
I20260927 07:08:04.399052 5598 consensus_queue.cc:260] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:04.399351 5598 ts_tablet_manager.cc:1428] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260927 07:08:04.433523 5708 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:04.389251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:58783)
0927 07:08:04.389465 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.389505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:04.389989 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:04.433326 (+ 43337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
W20260927 07:08:04.433818 5706 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40959 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:40959: Client connection negotiation failed: client connection to 127.21.2.126:40959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:08:04.434186 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.434069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:50707)
0927 07:08:04.434126 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.434160 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 07:08:04.434615 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.434484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:39785)
0927 07:08:04.434537 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.434572 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:08:04.434823 5706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:08:04.481379 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:04.481573 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:04.481598 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:04.482672 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:04.482707 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:04.482733 5710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:04.484107 5710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:46477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=40149
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:04.484292 5710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:04.484452 5710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:04.489199 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.489210 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:04.489384 5710 server_base.cc:1034] running on GCE node
W20260927 07:08:04.489207 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:04.489673 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:04.489930 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:04.491115 5710 hybrid_clock.cc:648] HybridClock initialized: now 1790492884491090 us; error 50 us; skew 500 ppm
Sep 27 07:08:04 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:04.493641 5710 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:08:04.494037 5710 webserver.cc:460] Webserver started at http://127.21.2.66:40149/ using document root <none> and password file <none>
I20260927 07:08:04.494228 5710 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:04.494268 5710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:04.495553 5710 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:04.496106 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.496284 5710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.496384 5710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "926cddd481dc4140a6926eef415d1f3b"
format_stamp: "Formatted at 2026-09-27 07:07:22 on dist-test-slave-ltqz"
server_key: "fdce70011149f28a8ae4bf7b4333282d"
server_key_iv: "ca56076e33d1e0c5140d70d8c717cb6b"
server_key_version: "encryptionkey@0"
I20260927 07:08:04.496714 5710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:04.507786 5710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:04.509671 5710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:04.509833 5710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:04.510063 5710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:04.510375 5710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:04.510432 5710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.510483 5710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:04.510511 5710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.516559 5710 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:46477
I20260927 07:08:04.516642 5839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:46477 every 8 connection(s)
I20260927 07:08:04.516990 5710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:04.518046 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.517692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:41769)
0927 07:08:04.517841 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.517908 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260927 07:08:04.518460 5840 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40959 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:40959: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
I20260927 07:08:04.518729 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.518642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:40875)
0927 07:08:04.518687 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.518712 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 07:08:04.518956 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.518907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:37603)
0927 07:08:04.518928 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.518945 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 07:08:04.519059 5840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:08:04.522535 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5710
I20260927 07:08:04.522802 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:43857
--local_ip_for_outbound_sockets=127.21.2.67
--tserver_master_addrs=127.21.2.126:40959
--webserver_port=33607
--webserver_interface=127.21.2.67
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:04.602988 5843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:04.603200 5843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:04.603224 5843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:04.604399 5843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:04.604442 5843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:04.604470 5843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:04.605937 5843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=33607
--tserver_master_addrs=127.21.2.126:40959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:04.606117 5843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:04.606282 5843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:04.611083 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.611064 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:04.611105 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:04.611420 5843 server_base.cc:1034] running on GCE node
I20260927 07:08:04.611639 5843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:04.611889 5843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:04.613083 5843 hybrid_clock.cc:648] HybridClock initialized: now 1790492884613070 us; error 30 us; skew 500 ppm
Sep 27 07:08:04 dist-test-slave-ltqz krb5kdc[4407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:04.615698 5843 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:08:04.616155 5843 webserver.cc:460] Webserver started at http://127.21.2.67:33607/ using document root <none> and password file <none>
I20260927 07:08:04.616386 5843 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:04.616456 5843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:04.618607 5843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:08:04.619467 5858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.619647 5843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:04.619750 5843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2"
format_stamp: "Formatted at 2026-09-27 07:07:23 on dist-test-slave-ltqz"
server_key: "47e1171263d9cd15390509f9bf783b9f"
server_key_iv: "d8c4246491633530a5763eb795e057a8"
server_key_version: "encryptionkey@0"
I20260927 07:08:04.620116 5843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:04.638183 5843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:04.640239 5843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:04.640403 5843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:04.640700 5843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:04.641021 5843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:04.641085 5843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.641129 5843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:04.641156 5843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:04.647733 5843 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:43857
I20260927 07:08:04.647773 5971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:43857 every 8 connection(s)
I20260927 07:08:04.648144 5843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:04.649225 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.648859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:46287)
0927 07:08:04.649012 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.649090 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
W20260927 07:08:04.649626 5972 heartbeater.cc:644] Failed to heartbeat to 127.21.2.126:40959 (0 consecutive failures): Network error: Failed to ping master at 127.21.2.126:40959: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
I20260927 07:08:04.649917 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.649811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:58435)
0927 07:08:04.649863 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.649888 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:08:04.650192 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:04.650109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:49933)
0927 07:08:04.650152 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:04.650173 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:08:04.650300 5972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:08:04.657940 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 5843
I20260927 07:08:05.435487 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:05.435199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:47977)
0927 07:08:05.435344 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:05.435437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260927 07:08:05.519603 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:05.519352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:42085)
0927 07:08:05.519514 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:05.519547 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260927 07:08:05.650784 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:05.650591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:33765)
0927 07:08:05.650704 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:05.650734 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260927 07:08:06.177829 5979 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:06.177948 5979 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:06.178334 5979 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
Sep 27 07:08:06 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.67@KRBTEST.COM
Sep 27 07:08:06 dist-test-slave-ltqz krb5kdc[4407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492884, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.66@KRBTEST.COM
I20260927 07:08:06.188475 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.178478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:43857 (local address 127.21.2.65:49829)
0927 07:08:06.178615 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:06.178627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:06.178690 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:06.179600 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:06.179601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:06.179828 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:06.180008 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:06.180013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.181112 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.181113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:06.181887 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:06.181889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.182506 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.182507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:06.182602 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:06.183241 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:06.183253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:06.184724 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:06.186734 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:06.186741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:06.186744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:06.186968 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:06.187281 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:06.187282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:06.187447 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:06.187953 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:06.187959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:06.188078 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 07:08:06.188163 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 07:08:06.188348 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":59,"thread_start_us":48,"threads_started":1}
I20260927 07:08:06.188488 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.178487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:46477 (local address 127.21.2.65:46599)
0927 07:08:06.178630 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:06.178639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:06.178690 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:06.179492 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:06.179495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:06.179839 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:06.180008 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:06.180014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.181098 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.181098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:06.181887 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:06.181889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.182478 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.182479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:06.182609 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:06.183195 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:06.183203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:06.185086 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:06.187115 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:06.187116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:06.187324 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:06.187752 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:06.187753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:06.187804 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:06.188217 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:06.188217 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:06.188255 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:08:06.188325 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:08:06.188399 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 07:08:06.188643 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.178605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:46599 (local address 127.21.2.66:46477)
0927 07:08:06.178811 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:06.178818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:06.178836 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:08:06.179243 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:06.179245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:06.179286 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:06.179458 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:06.180130 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.180989 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.181985 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.182385 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.182515 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:06.185215 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:06.185219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:06.185219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:06.185231 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:06.187001 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187419 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:06.187423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:06.187423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:06.187677 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187864 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:06.187865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:06.187865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:06.188124 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:06.188221 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:06.188391 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 07:08:06.188510 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":60,"threads_started":1}
I20260927 07:08:06.188833 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.178653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:49829 (local address 127.21.2.67:43857)
0927 07:08:06.178770 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:06.178781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:06.178811 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:08:06.179243 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:06.179245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:06.179286 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:06.179454 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:06.180141 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.180993 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.182004 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:06.182402 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:06.182531 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:06.184854 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:06.184859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:06.184859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:06.184872 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:06.186638 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187071 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:06.187077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:06.187081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:06.187164 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:06.187530 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:06.187530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:06.187531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:06.187880 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:06.187966 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:06.188249 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 07:08:06.188410 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
W20260927 07:08:06.189625 5593 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
W20260927 07:08:06.189719 5595 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
I20260927 07:08:06.189765 5595 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c4f47b98e24f6cae85fbaa9a37ab63; no voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b
I20260927 07:08:06.189873 5979 raft_consensus.cc:2726] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:08:06.436101 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.435952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:36649)
0927 07:08:06.436022 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:06.436060 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 07:08:06.520237 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.520015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:40391)
0927 07:08:06.520145 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:06.520193 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260927 07:08:06.651517 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:06.651271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:56133)
0927 07:08:06.651428 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:06.651476 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260927 07:08:07.436874 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:07.436589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:43377)
0927 07:08:07.436790 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:07.436830 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":62,"threads_started":1}
I20260927 07:08:07.520888 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:07.520662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:33489)
0927 07:08:07.520778 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:07.520815 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260927 07:08:07.652187 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:07.651933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:47143)
0927 07:08:07.652062 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:07.652141 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260927 07:08:08.228114 5989 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:08.228222 5989 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:08.228482 5989 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
W20260927 07:08:08.228986 5595 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
W20260927 07:08:08.229081 5593 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
I20260927 07:08:08.229148 5593 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c4f47b98e24f6cae85fbaa9a37ab63; no voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b
I20260927 07:08:08.229242 5989 raft_consensus.cc:2726] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:08:08.437613 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:08.437363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:35489)
0927 07:08:08.437510 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:08.437547 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260927 07:08:08.521651 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:08.521424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:51533)
0927 07:08:08.521544 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:08.521586 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260927 07:08:08.652863 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:08.652659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:48665)
0927 07:08:08.652776 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:08.652815 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 07:08:09.438489 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:09.438236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:55753)
0927 07:08:09.438399 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:09.438443 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260927 07:08:09.522308 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:09.522086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:57209)
0927 07:08:09.522225 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:09.522266 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260927 07:08:09.653607 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:09.653389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:37199)
0927 07:08:09.653528 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:09.653564 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260927 07:08:10.167337 5996 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:10.167510 5996 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:10.167680 5996 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
W20260927 07:08:10.168080 5595 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
W20260927 07:08:10.168184 5593 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
I20260927 07:08:10.168219 5593 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c4f47b98e24f6cae85fbaa9a37ab63; no voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b
I20260927 07:08:10.168311 5996 raft_consensus.cc:2726] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:08:10.439210 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:10.438969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:54771)
0927 07:08:10.439122 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:10.439163 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 07:08:10.523012 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:10.522795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:57701)
0927 07:08:10.522928 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:10.522967 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 07:08:10.654274 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:10.654043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:60421)
0927 07:08:10.654194 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:10.654232 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260927 07:08:11.439960 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:11.439727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:35537)
0927 07:08:11.439869 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:11.439909 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260927 07:08:11.523772 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:11.523545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:59423)
0927 07:08:11.523680 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:11.523727 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260927 07:08:11.654948 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:11.654743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:58911)
0927 07:08:11.654871 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:11.654907 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260927 07:08:11.877760 6003 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:11.877856 6003 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:11.878099 6003 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
W20260927 07:08:11.878506 5593 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
W20260927 07:08:11.878602 5595 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
I20260927 07:08:11.878677 5595 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c4f47b98e24f6cae85fbaa9a37ab63; no voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b
I20260927 07:08:11.878760 6003 raft_consensus.cc:2726] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:08:12.440694 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:12.440454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:44269)
0927 07:08:12.440607 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:12.440647 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260927 07:08:12.524425 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:12.524207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:48781)
0927 07:08:12.524342 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:12.524380 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 07:08:12.655560 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:12.655314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:49053)
0927 07:08:12.655488 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:12.655519 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260927 07:08:13.441447 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:13.441197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:54847)
0927 07:08:13.441357 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:13.441399 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260927 07:08:13.525152 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:13.524913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:34533)
0927 07:08:13.525051 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:13.525095 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260927 07:08:13.656226 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:13.656004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:47845)
0927 07:08:13.656147 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:13.656179 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260927 07:08:14.442198 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:14.441938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:52819)
0927 07:08:14.442079 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:14.442141 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260927 07:08:14.525969 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:14.525726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:54995)
0927 07:08:14.525879 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:14.525919 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260927 07:08:14.657115 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:14.656825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:50149)
0927 07:08:14.657035 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:14.657071 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260927 07:08:15.442818 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:15.442610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:57225)
0927 07:08:15.442737 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:15.442773 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260927 07:08:15.526667 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:15.526467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:48361)
0927 07:08:15.526589 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:15.526624 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260927 07:08:15.658030 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:15.657784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:44153)
0927 07:08:15.657944 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:15.657983 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260927 07:08:16.443565 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:16.443293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:43123)
0927 07:08:16.443469 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:16.443512 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260927 07:08:16.527464 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:16.527171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:55175)
0927 07:08:16.527327 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:16.527416 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260927 07:08:16.658759 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:16.658534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:57335)
0927 07:08:16.658673 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:16.658711 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260927 07:08:17.444259 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:17.444014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:56733)
0927 07:08:17.444167 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:17.444210 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260927 07:08:17.528108 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:17.527895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:33269)
0927 07:08:17.528030 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:17.528065 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 07:08:17.659523 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:17.659269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:48033)
0927 07:08:17.659439 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:17.659472 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260927 07:08:17.715106 6022 raft_consensus.cc:482] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:17.715224 6022 raft_consensus.cc:504] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff79a2cc8440d8ad0fc0a2ec6fc7f2" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 43857 } } peers { permanent_uuid: "926cddd481dc4140a6926eef415d1f3b" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 46477 } } peers { permanent_uuid: "e2c4f47b98e24f6cae85fbaa9a37ab63" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 36743 } }
I20260927 07:08:17.715482 6022 leader_election.cc:288] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857), 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477)
W20260927 07:08:17.715896 5595 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77ff79a2cc8440d8ad0fc0a2ec6fc7f2 (127.21.2.67:43857): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
W20260927 07:08:17.715994 5593 leader_election.cc:341] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 926cddd481dc4140a6926eef415d1f3b (127.21.2.66:46477): Not found: Tablet not found: e1e9e459488842a495a2fa479cfb101a
I20260927 07:08:17.716054 5593 leader_election.cc:302] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c4f47b98e24f6cae85fbaa9a37ab63; no voters: 77ff79a2cc8440d8ad0fc0a2ec6fc7f2, 926cddd481dc4140a6926eef415d1f3b
I20260927 07:08:17.716157 6022 raft_consensus.cc:2726] T e1e9e459488842a495a2fa479cfb101a P e2c4f47b98e24f6cae85fbaa9a37ab63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:08:18.444940 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:18.444730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:52001)
0927 07:08:18.444851 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:18.444892 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260927 07:08:18.528791 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:18.528573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:52129)
0927 07:08:18.528705 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:18.528745 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260927 07:08:18.660254 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:18.660032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:42479)
0927 07:08:18.660165 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:18.660208 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260927 07:08:19.445981 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:19.445741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.65:47469)
0927 07:08:19.445902 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:19.445936 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260927 07:08:19.529702 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:19.529466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.66:54657)
0927 07:08:19.529613 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:19.529654 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260927 07:08:19.661073 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:19.660806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40959 (local address 127.21.2.67:37007)
0927 07:08:19.660968 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:19.661015 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.126:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260927 07:08:19.663038 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5545
I20260927 07:08:19.667897 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5710
I20260927 07:08:19.672463 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 5843
2026-09-27T07:08:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58362 ms)
[----------] 30 tests from SecurityITest (257975 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:19 dist-test-slave-ltqz krb5kdc[6031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:19 dist-test-slave-ltqz krb5kdc[6031](info): set up 2 sockets
Sep 27 07:08:19 dist-test-slave-ltqz krb5kdc[6031](info): commencing operation
krb5kdc: starting...
W20260927 07:08:20.723150 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:20 dist-test-slave-ltqz krb5kdc[6031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:20.842142 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34581
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:20.921980 6044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:20.922238 6044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:20.922276 6044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:20.923454 6044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:20.923525 6044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:20.923549 6044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:20.923570 6044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:20.923600 6044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:20.925077 6044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:20.925355 6044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:20.925577 6044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:20.930261 6049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:20.930294 6050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:20.930294 6052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:20.930603 6044 server_base.cc:1034] running on GCE node
I20260927 07:08:20.930799 6044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:20.931046 6044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:20.932214 6044 hybrid_clock.cc:648] HybridClock initialized: now 1790492900932203 us; error 26 us; skew 500 ppm
Sep 27 07:08:20 dist-test-slave-ltqz krb5kdc[6031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492900, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:20.934564 6044 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:20.934985 6044 webserver.cc:460] Webserver started at http://127.21.2.126:46295/ using document root <none> and password file <none>
I20260927 07:08:20.935209 6044 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:20.935253 6044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:20.935432 6044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:20.936467 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "4b2fe88e9bc1400fae7fcc08282db3e9"
format_stamp: "Formatted at 2026-09-27 07:08:20 on dist-test-slave-ltqz"
server_key: "88b3cc6d2b16d13430d33626646c56a4"
server_key_iv: "9a82860a21bcc2d2d53c658f645a07cc"
server_key_version: "encryptionkey@0"
I20260927 07:08:20.936822 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "4b2fe88e9bc1400fae7fcc08282db3e9"
format_stamp: "Formatted at 2026-09-27 07:08:20 on dist-test-slave-ltqz"
server_key: "88b3cc6d2b16d13430d33626646c56a4"
server_key_iv: "9a82860a21bcc2d2d53c658f645a07cc"
server_key_version: "encryptionkey@0"
I20260927 07:08:20.938145 6044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:20.938835 6059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:20.939025 6044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:20.939110 6044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "4b2fe88e9bc1400fae7fcc08282db3e9"
format_stamp: "Formatted at 2026-09-27 07:08:20 on dist-test-slave-ltqz"
server_key: "88b3cc6d2b16d13430d33626646c56a4"
server_key_iv: "9a82860a21bcc2d2d53c658f645a07cc"
server_key_version: "encryptionkey@0"
I20260927 07:08:20.939163 6044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:20.952111 6044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:20.954201 6044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:20.954346 6044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:20.958206 6044 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34581
I20260927 07:08:20.958240 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34581 every 8 connection(s)
I20260927 07:08:20.958619 6044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:20.959568 6112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:20.961817 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: Bootstrap starting.
I20260927 07:08:20.962407 6112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:20.962697 6112 log.cc:824] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:20.963435 6112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: No bootstrap required, opened a new log
I20260927 07:08:20.965013 6112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } }
I20260927 07:08:20.965148 6112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:20.965196 6112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2fe88e9bc1400fae7fcc08282db3e9, State: Initialized, Role: FOLLOWER
I20260927 07:08:20.965341 6112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } }
I20260927 07:08:20.965405 6112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:20.965425 6112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:20.965494 6112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:20.966015 6112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } }
I20260927 07:08:20.966125 6112 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2fe88e9bc1400fae7fcc08282db3e9; no voters:
I20260927 07:08:20.966313 6112 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:20.966349 6115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:20.966495 6115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4b2fe88e9bc1400fae7fcc08282db3e9, State: Running, Role: LEADER
I20260927 07:08:20.966567 6112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:20.966631 6115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } }
I20260927 07:08:20.967028 6117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2fe88e9bc1400fae7fcc08282db3e9. Latest consensus state: current_term: 1 leader_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } } }
I20260927 07:08:20.967161 6116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fe88e9bc1400fae7fcc08282db3e9" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34581 } } }
I20260927 07:08:20.967206 6117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:20.967245 6116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:20.967571 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6044
I20260927 07:08:20.967684 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:20.967882 21513 external_mini_cluster.cc:1428] Setting key a299e647013cfb1e1af91c0c4e467c8e
I20260927 07:08:20.968142 6125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:20.968706 6125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:20.970908 6125 catalog_manager.cc:1347] Generated new cluster ID: c36c813ec1de4b4397803de59b870b09
I20260927 07:08:20.970978 6125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:20 dist-test-slave-ltqz krb5kdc[6031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:20.976572 6125 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:20.977278 6125 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:20.979339 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:20.968475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.21.2.126:34581)
0927 07:08:20.968708 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:20.968717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:20.968752 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:08:20.968881 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:20.968888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:20.968944 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:20.969115 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:20.969833 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:20.970794 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:20.971930 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:20.972596 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:20.972696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:20.975614 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:20.975618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:20.975618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:20.975630 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:20.977749 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:20.978154 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:20.978157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:20.978157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:20.978259 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:20.978469 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:20.978470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:20.978470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:20.978727 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:20.978871 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:20.978988 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 07:08:20.979130 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
I20260927 07:08:20.984484 6125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: Generated new TSK 0
I20260927 07:08:20.984650 6125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:08:21.014182 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:21.096606 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:21.096815 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:21.096839 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:21.097920 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:21.097959 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:21.097988 6138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:21.099501 6138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:21.099701 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:21.099887 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:21.104853 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.104868 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.104864 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.105096 6138 server_base.cc:1034] running on GCE node
I20260927 07:08:21.105331 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:21.105587 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:21.106752 6138 hybrid_clock.cc:648] HybridClock initialized: now 1790492901106741 us; error 30 us; skew 500 ppm
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:21.109304 6138 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:08:21.109684 6138 webserver.cc:460] Webserver started at http://127.21.2.65:41323/ using document root <none> and password file <none>
I20260927 07:08:21.109880 6138 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:21.109920 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:21.110006 6138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:21.110980 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "20b7f6a079914b06a46d22e3a36cea29"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "fb58a4433e1be42b8f2a9c305681fac3"
server_key_iv: "ce3c197594db3f3d766273fb1b93d3d1"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.111284 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "20b7f6a079914b06a46d22e3a36cea29"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "fb58a4433e1be42b8f2a9c305681fac3"
server_key_iv: "ce3c197594db3f3d766273fb1b93d3d1"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.112623 6138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:21.113312 6153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.113523 6138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:21.113623 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "20b7f6a079914b06a46d22e3a36cea29"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "fb58a4433e1be42b8f2a9c305681fac3"
server_key_iv: "ce3c197594db3f3d766273fb1b93d3d1"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.113703 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:21.130568 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:21.132977 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:21.133126 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:21.133378 6138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:21.133682 6138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:21.133714 6138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.133755 6138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:21.133782 6138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.139344 6138 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37369
I20260927 07:08:21.139447 6266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37369 every 8 connection(s)
I20260927 07:08:21.139765 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:21.149089 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6138
I20260927 07:08:21.149181 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.140508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54891 (local address 127.21.2.126:34581)
0927 07:08:21.140553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.140555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.140781 (+ 226us) server_negotiation.cc:407] Connection header received
0927 07:08:21.141306 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.141307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.141321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.141375 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.142386 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.142957 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.143985 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.144192 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.144269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.146456 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.146458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.146458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:21.146469 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.147760 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.148164 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.148165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.148166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.148228 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.148488 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.148489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.148489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.148606 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.148684 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.148974 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.149080 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:08:21.149215 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:21.149362 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.140464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34581 (local address 127.21.2.65:54891)
0927 07:08:21.140651 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.140686 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.141197 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.141444 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.141448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.142077 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:21.142266 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.142272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.143049 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.143051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.143876 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.143879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.144276 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.144278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.144412 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.144872 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:21.144880 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:21.146351 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:21.147854 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.147861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.147864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.148084 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.148303 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.148303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.148304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.148413 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.148675 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:21.148682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:21.148827 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.148940 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.149104 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260927 07:08:21.149427 21513 external_mini_cluster.cc:1428] Setting key d1728e691431ce01a500b61a7cabd0e9
I20260927 07:08:21.149909 6267 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34581
I20260927 07:08:21.149994 6267 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:21.150218 6267 heartbeater.cc:511] Master 127.21.2.126:34581 requested a full tablet report, sending...
I20260927 07:08:21.150768 6075 ts_manager.cc:194] Registered new tserver with Master: 20b7f6a079914b06a46d22e3a36cea29 (127.21.2.65:37369)
I20260927 07:08:21.151438 6075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:54891
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:08:21.173866 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:21.258925 6272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:21.259132 6272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:21.259157 6272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:21.260382 6272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:21.260434 6272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:21.260462 6272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:21.261956 6272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:21.262214 6272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:21.262491 6272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:21.267740 6278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.267765 6277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.267795 6280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.268069 6272 server_base.cc:1034] running on GCE node
I20260927 07:08:21.268344 6272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:21.268649 6272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:21.269802 6272 hybrid_clock.cc:648] HybridClock initialized: now 1790492901269786 us; error 30 us; skew 500 ppm
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:21.272361 6272 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:08:21.272756 6272 webserver.cc:460] Webserver started at http://127.21.2.66:42423/ using document root <none> and password file <none>
I20260927 07:08:21.272953 6272 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:21.272993 6272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:21.273079 6272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:21.274200 6272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "41c63bb82df844cb94b70282f950bc69"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e7a1ca40a3e955590b992824e9f3ac34"
server_key_iv: "dfe8fba9a7d6824166b3839a407a2e5a"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.274521 6272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "41c63bb82df844cb94b70282f950bc69"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e7a1ca40a3e955590b992824e9f3ac34"
server_key_iv: "dfe8fba9a7d6824166b3839a407a2e5a"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.276340 6272 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:08:21.277484 6287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.277760 6272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:21.277848 6272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "41c63bb82df844cb94b70282f950bc69"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e7a1ca40a3e955590b992824e9f3ac34"
server_key_iv: "dfe8fba9a7d6824166b3839a407a2e5a"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.277933 6272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:21.301781 6272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:21.304191 6272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:21.304368 6272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:21.304611 6272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:21.304956 6272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:21.305012 6272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.305054 6272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:21.305083 6272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.310889 6272 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:37723
I20260927 07:08:21.310930 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:37723 every 8 connection(s)
I20260927 07:08:21.311241 6272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:21.318778 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6272
I20260927 07:08:21.318908 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:21.319133 21513 external_mini_cluster.cc:1428] Setting key cd8be06a89c37f7321b3020ec3d9861e
I20260927 07:08:21.320606 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.311965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:59889 (local address 127.21.2.126:34581)
0927 07:08:21.312015 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.312017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.312267 (+ 250us) server_negotiation.cc:407] Connection header received
0927 07:08:21.312731 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.312732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.312743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.312797 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.313693 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.314192 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.315129 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.315312 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.315397 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.317678 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.317680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.317681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:21.317692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.319088 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.319590 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.319592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.319592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.319656 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.319951 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.319952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.319953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.320079 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.320151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.320430 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.320520 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:08:21.320781 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.311962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34581 (local address 127.21.2.66:59889)
0927 07:08:21.312149 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.312185 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.312650 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.312876 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.312879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.313434 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:21.313618 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.313622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.314270 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.314271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.315057 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.315059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.315444 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.315445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.315567 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.316105 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:21.316114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:21.317577 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:21.319165 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.319185 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.319188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.319497 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.319718 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.319718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.319719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.319871 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.320153 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:21.320160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:21.320292 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.320395 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.320552 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260927 07:08:21.321697 6401 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34581
I20260927 07:08:21.321807 6401 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:21.322013 6401 heartbeater.cc:511] Master 127.21.2.126:34581 requested a full tablet report, sending...
I20260927 07:08:21.322398 6075 ts_manager.cc:194] Registered new tserver with Master: 41c63bb82df844cb94b70282f950bc69 (127.21.2.66:37723)
I20260927 07:08:21.322935 6075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:59889
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:08:21.344177 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:21.425319 6406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:21.425532 6406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:21.425557 6406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:21.426745 6406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:21.426792 6406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:21.426819 6406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:21.428336 6406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:21.428588 6406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:21.428843 6406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:21.433820 6411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.433810 6412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.433810 6414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.434003 6406 server_base.cc:1034] running on GCE node
I20260927 07:08:21.434300 6406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:21.434592 6406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:21.435752 6406 hybrid_clock.cc:648] HybridClock initialized: now 1790492901435744 us; error 50 us; skew 500 ppm
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:21.439021 6406 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:08:21.439600 6406 webserver.cc:460] Webserver started at http://127.21.2.67:38771/ using document root <none> and password file <none>
I20260927 07:08:21.439842 6406 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:21.439910 6406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:21.440047 6406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:21.441166 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "f39c8cbf952348ad86399d29e5d8d13e"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e28d645097237a891bc684aba862d48a"
server_key_iv: "9742fea59f9778feaba26512cb109ec6"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.441625 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "f39c8cbf952348ad86399d29e5d8d13e"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e28d645097237a891bc684aba862d48a"
server_key_iv: "9742fea59f9778feaba26512cb109ec6"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.443101 6406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:21.443943 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.444128 6406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.444238 6406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "f39c8cbf952348ad86399d29e5d8d13e"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "e28d645097237a891bc684aba862d48a"
server_key_iv: "9742fea59f9778feaba26512cb109ec6"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.444346 6406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:21.463593 6406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:21.466008 6406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:21.466181 6406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:21.466463 6406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:21.466817 6406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:21.466881 6406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.466924 6406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:21.466951 6406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.473140 6406 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:41291
I20260927 07:08:21.473193 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:41291 every 8 connection(s)
I20260927 07:08:21.473543 6406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:08:21 dist-test-slave-ltqz krb5kdc[6031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492901, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:21.479528 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6406
I20260927 07:08:21.479633 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:21.479823 21513 external_mini_cluster.cc:1428] Setting key c8a74e7abd0950a331ecae818248fea0
I20260927 07:08:21.482858 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.474287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:58677 (local address 127.21.2.126:34581)
0927 07:08:21.474361 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.474363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.474549 (+ 186us) server_negotiation.cc:407] Connection header received
0927 07:08:21.475062 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.475063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.475073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.475135 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.476130 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.476622 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.477574 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.477749 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.477798 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.480086 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.480087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.480088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:21.480096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.481610 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.482026 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.482027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.482028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.482092 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.482331 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.482333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.482333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.482458 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.482523 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.482729 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.482792 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:08:21.483098 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.474266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34581 (local address 127.21.2.67:58677)
0927 07:08:21.474412 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.474453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.474976 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.475185 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.475188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.475852 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:21.476045 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.476050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.476705 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.476705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.477503 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.477505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.477819 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.477819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.477916 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.478457 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:21.478466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:21.479985 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:21.481684 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.481690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.481691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.481960 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.482152 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:21.482153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:21.482153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:21.482249 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:21.482523 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:21.482527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:21.482635 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.482715 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.482837 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260927 07:08:21.483629 6535 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34581
I20260927 07:08:21.483712 6535 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:21.483861 6535 heartbeater.cc:511] Master 127.21.2.126:34581 requested a full tablet report, sending...
I20260927 07:08:21.484174 6075 ts_manager.cc:194] Registered new tserver with Master: f39c8cbf952348ad86399d29e5d8d13e (127.21.2.67:41291)
I20260927 07:08:21.484504 6075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:58677
I20260927 07:08:21.491904 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:21.499655 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.493282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.21.2.126:34581)
0927 07:08:21.493358 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.493361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.493439 (+ 78us) server_negotiation.cc:407] Connection header received
0927 07:08:21.493540 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.493541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.493556 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.493636 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.494238 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.494781 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.495704 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.495940 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.496025 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.497269 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.497270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.497271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:21.497278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.498429 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.498765 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.498766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.498767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.498826 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:21.499024 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:21.499025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:21.499025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:21.499175 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.499284 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.499465 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.499550 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:08:21.501322 6075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:21.502866 6075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:21.509377 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.505192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:41291 (local address 127.0.0.1:37024)
0927 07:08:21.505630 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.505641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.505704 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.505955 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.505955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.505957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:21.506005 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.506007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.507243 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.507244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.508345 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.508347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.509011 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.509097 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509111 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.509214 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.509277 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":71,"threads_started":1}
I20260927 07:08:21.509454 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.505350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.21.2.65:37369)
0927 07:08:21.505563 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.505567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.505793 (+ 226us) server_negotiation.cc:407] Connection header received
0927 07:08:21.505843 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.505845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.505859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.505925 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:21.506106 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.506822 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.508421 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.509093 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509160 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509213 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.509307 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.509359 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260927 07:08:21.509578 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.505094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.21.2.66:37723)
0927 07:08:21.505303 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.505308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.505320 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:08:21.505375 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.505377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.505394 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.505482 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:21.505656 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.506838 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.508297 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.509023 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509098 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509157 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.509432 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.509483 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260927 07:08:21.509933 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.505305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.21.2.67:41291)
0927 07:08:21.505493 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.505498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.505698 (+ 200us) server_negotiation.cc:407] Connection header received
0927 07:08:21.505771 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.505774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.505790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.505877 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:21.506071 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.507143 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.508435 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.508908 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509009 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509070 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.509832 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.509874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260927 07:08:21.509445 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.504998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37369 (local address 127.0.0.1:40436)
0927 07:08:21.505569 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.505579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.505629 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.506008 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.506009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.506011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:21.506051 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.506052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.506933 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.506934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.508307 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.508310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.509180 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.509242 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509244 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.509293 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.509326 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260927 07:08:21.509548 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.504987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37723 (local address 127.0.0.1:41082)
0927 07:08:21.505140 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.505184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.505250 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.505488 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.505491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.505493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:21.505562 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.505564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.506958 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.506959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.508206 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.508209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.509201 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.509202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.509264 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.509268 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.509318 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.509360 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260927 07:08:21.510754 6201 tablet_service.cc:1467] Processing CreateTablet for tablet a8348794f6b84f129697bd87eb2a5644 (DEFAULT_TABLE table=test-table [id=54aaa19c95ef4ef3af0b5752b24dcec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:21.510794 6335 tablet_service.cc:1467] Processing CreateTablet for tablet a8348794f6b84f129697bd87eb2a5644 (DEFAULT_TABLE table=test-table [id=54aaa19c95ef4ef3af0b5752b24dcec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:21.510854 6469 tablet_service.cc:1467] Processing CreateTablet for tablet a8348794f6b84f129697bd87eb2a5644 (DEFAULT_TABLE table=test-table [id=54aaa19c95ef4ef3af0b5752b24dcec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:21.511116 6201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8348794f6b84f129697bd87eb2a5644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:21.511154 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8348794f6b84f129697bd87eb2a5644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:21.511154 6335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8348794f6b84f129697bd87eb2a5644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:21.513697 6552 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: Bootstrap starting.
I20260927 07:08:21.514429 6554 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Bootstrap starting.
I20260927 07:08:21.514644 6552 tablet_bootstrap.cc:654] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:21.515112 6554 tablet_bootstrap.cc:654] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:21.515272 6552 log.cc:824] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:21.515733 6554 log.cc:824] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:21.516074 6553 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Bootstrap starting.
I20260927 07:08:21.516216 6552 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: No bootstrap required, opened a new log
I20260927 07:08:21.516283 6552 ts_tablet_manager.cc:1397] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:08:21.516695 6553 tablet_bootstrap.cc:654] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:21.517194 6553 log.cc:824] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:21.517712 6554 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: No bootstrap required, opened a new log
I20260927 07:08:21.517786 6554 ts_tablet_manager.cc:1397] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 07:08:21.517904 6553 tablet_bootstrap.cc:492] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: No bootstrap required, opened a new log
I20260927 07:08:21.518011 6553 ts_tablet_manager.cc:1397] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:21.518517 6552 raft_consensus.cc:348] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.518700 6552 raft_consensus.cc:374] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:21.518738 6552 raft_consensus.cc:729] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b7f6a079914b06a46d22e3a36cea29, State: Initialized, Role: FOLLOWER
I20260927 07:08:21.518883 6552 consensus_queue.cc:260] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.519167 6552 ts_tablet_manager.cc:1428] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:21.519320 6267 heartbeater.cc:503] Master 127.21.2.126:34581 was elected leader, sending a full tablet report...
I20260927 07:08:21.519817 6554 raft_consensus.cc:348] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.519960 6554 raft_consensus.cc:374] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:21.520004 6554 raft_consensus.cc:729] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c63bb82df844cb94b70282f950bc69, State: Initialized, Role: FOLLOWER
I20260927 07:08:21.519912 6553 raft_consensus.cc:348] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.520094 6553 raft_consensus.cc:374] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:21.520148 6553 raft_consensus.cc:729] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39c8cbf952348ad86399d29e5d8d13e, State: Initialized, Role: FOLLOWER
I20260927 07:08:21.520143 6554 consensus_queue.cc:260] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.520262 6553 consensus_queue.cc:260] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.520430 6554 ts_tablet_manager.cc:1428] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:08:21.520488 6401 heartbeater.cc:503] Master 127.21.2.126:34581 was elected leader, sending a full tablet report...
I20260927 07:08:21.520569 6553 ts_tablet_manager.cc:1428] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:21.520615 6535 heartbeater.cc:503] Master 127.21.2.126:34581 was elected leader, sending a full tablet report...
W20260927 07:08:21.561820 6402 tablet.cc:2369] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:21.591614 6559 raft_consensus.cc:482] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:21.591763 6559 raft_consensus.cc:504] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.592228 6559 leader_election.cc:288] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b7f6a079914b06a46d22e3a36cea29 (127.21.2.65:37369), f39c8cbf952348ad86399d29e5d8d13e (127.21.2.67:41291)
I20260927 07:08:21.596195 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.592687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47313 (local address 127.21.2.67:41291)
0927 07:08:21.592737 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.592739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.592879 (+ 140us) server_negotiation.cc:407] Connection header received
0927 07:08:21.592906 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.592907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.592921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.592970 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:21.593141 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.593804 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595203 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.595822 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595891 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.595904 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.596080 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.596121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:08:21.596195 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.592532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:40005 (local address 127.21.2.65:37369)
0927 07:08:21.592608 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.592610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.592625 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:08:21.592705 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.592706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.592721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.592778 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:21.592944 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.593701 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595209 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.595811 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595892 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.595905 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.596080 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.596121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 07:08:21.596197 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.592457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37369 (local address 127.21.2.66:40005)
0927 07:08:21.592544 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.592577 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.592643 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.592793 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.592794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.592796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:21.592882 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.592885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.593781 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.593782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.595107 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.595110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.595904 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.595989 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.595992 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.596052 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.596101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":99}
I20260927 07:08:21.596204 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.592582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:41291 (local address 127.21.2.66:47313)
0927 07:08:21.592765 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.592775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.592839 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.592994 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.592995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.592996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:21.593048 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.593049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.593877 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.593877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.595107 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.595110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.595905 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.595905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.595987 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.595992 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.596052 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.596101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":6,"thread_start_us":117,"threads_started":1}
I20260927 07:08:21.596597 6221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8348794f6b84f129697bd87eb2a5644" candidate_uuid: "41c63bb82df844cb94b70282f950bc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b7f6a079914b06a46d22e3a36cea29" is_pre_election: true
I20260927 07:08:21.596722 6221 raft_consensus.cc:2454] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c63bb82df844cb94b70282f950bc69 in term 0.
I20260927 07:08:21.596915 6289 leader_election.cc:302] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b7f6a079914b06a46d22e3a36cea29, 41c63bb82df844cb94b70282f950bc69; no voters:
I20260927 07:08:21.597069 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8348794f6b84f129697bd87eb2a5644" candidate_uuid: "41c63bb82df844cb94b70282f950bc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39c8cbf952348ad86399d29e5d8d13e" is_pre_election: true
I20260927 07:08:21.597087 6559 raft_consensus.cc:2781] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:21.597158 6559 raft_consensus.cc:482] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:21.597173 6559 raft_consensus.cc:3037] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:21.597214 6489 raft_consensus.cc:2454] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41c63bb82df844cb94b70282f950bc69 in term 0.
I20260927 07:08:21.597994 6559 raft_consensus.cc:504] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.598140 6559 leader_election.cc:288] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [CANDIDATE]: Term 1 election: Requested vote from peers 20b7f6a079914b06a46d22e3a36cea29 (127.21.2.65:37369), f39c8cbf952348ad86399d29e5d8d13e (127.21.2.67:41291)
I20260927 07:08:21.598286 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8348794f6b84f129697bd87eb2a5644" candidate_uuid: "41c63bb82df844cb94b70282f950bc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39c8cbf952348ad86399d29e5d8d13e"
I20260927 07:08:21.598331 6221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8348794f6b84f129697bd87eb2a5644" candidate_uuid: "41c63bb82df844cb94b70282f950bc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b7f6a079914b06a46d22e3a36cea29"
I20260927 07:08:21.598387 6489 raft_consensus.cc:3037] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:21.598415 6221 raft_consensus.cc:3037] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:21.599098 6221 raft_consensus.cc:2454] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c63bb82df844cb94b70282f950bc69 in term 1.
I20260927 07:08:21.599102 6489 raft_consensus.cc:2454] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41c63bb82df844cb94b70282f950bc69 in term 1.
I20260927 07:08:21.599264 6289 leader_election.cc:302] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c63bb82df844cb94b70282f950bc69, f39c8cbf952348ad86399d29e5d8d13e; no voters:
I20260927 07:08:21.599400 6559 raft_consensus.cc:2781] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:21.599586 6559 raft_consensus.cc:686] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 1 LEADER]: Becoming Leader. State: Replica: 41c63bb82df844cb94b70282f950bc69, State: Running, Role: LEADER
I20260927 07:08:21.599709 6559 consensus_queue.cc:237] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } }
I20260927 07:08:21.600577 6076 catalog_manager.cc:5495] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41c63bb82df844cb94b70282f950bc69 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "41c63bb82df844cb94b70282f950bc69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c63bb82df844cb94b70282f950bc69" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 } health_report { overall_health: UNKNOWN } } }
I20260927 07:08:21.625823 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.623400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.21.2.66:37723)
0927 07:08:21.623453 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.623455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.623465 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:21.623507 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.623508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.623529 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.623611 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:21.623750 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.624239 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.625035 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.625245 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.625330 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.625490 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:21.625597 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:21.625692 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.625711 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.625730 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:08:21.627669 6489 raft_consensus.cc:1261] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 1 FOLLOWER]: Refusing update from remote peer 41c63bb82df844cb94b70282f950bc69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:21.627674 6221 raft_consensus.cc:1261] T a8348794f6b84f129697bd87eb2a5644 P 20b7f6a079914b06a46d22e3a36cea29 [term 1 FOLLOWER]: Refusing update from remote peer 41c63bb82df844cb94b70282f950bc69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:21.628074 6562 consensus_queue.cc:1035] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39c8cbf952348ad86399d29e5d8d13e" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 41291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:21.628212 6559 consensus_queue.cc:1035] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b7f6a079914b06a46d22e3a36cea29" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:21.632942 6075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:21.633044 6075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:21.633780 6075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2fe88e9bc1400fae7fcc08282db3e9: Sending DeleteTablet for 3 replicas of tablet a8348794f6b84f129697bd87eb2a5644
I20260927 07:08:21.634143 6335 tablet_service.cc:1514] Processing DeleteTablet for tablet a8348794f6b84f129697bd87eb2a5644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:21 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 07:08:21.634155 6201 tablet_service.cc:1514] Processing DeleteTablet for tablet a8348794f6b84f129697bd87eb2a5644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:21 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:40436
I20260927 07:08:21.634361 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet a8348794f6b84f129697bd87eb2a5644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:21 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:37024
I20260927 07:08:21.634589 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6138
I20260927 07:08:21.635429 6577 tablet_replica.cc:330] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: stopping tablet replica
I20260927 07:08:21.635576 6577 raft_consensus.cc:2229] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:21.635753 6577 raft_consensus.cc:2258] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:21.636021 6575 tablet_replica.cc:330] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: stopping tablet replica
I20260927 07:08:21.636128 6575 raft_consensus.cc:2229] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:21.636224 6575 raft_consensus.cc:2258] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:21.636402 6577 ts_tablet_manager.cc:1905] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:21.636655 6575 ts_tablet_manager.cc:1905] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:21.638556 6575 ts_tablet_manager.cc:1918] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:21.638558 6577 ts_tablet_manager.cc:1918] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:21.638635 6575 log.cc:1196] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/a8348794f6b84f129697bd87eb2a5644
I20260927 07:08:21.638746 6577 log.cc:1196] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/a8348794f6b84f129697bd87eb2a5644
I20260927 07:08:21.638934 6575 ts_tablet_manager.cc:1939] T a8348794f6b84f129697bd87eb2a5644 P 41c63bb82df844cb94b70282f950bc69: Deleting consensus metadata
I20260927 07:08:21.639137 6577 ts_tablet_manager.cc:1939] T a8348794f6b84f129697bd87eb2a5644 P f39c8cbf952348ad86399d29e5d8d13e: Deleting consensus metadata
I20260927 07:08:21.639330 6060 catalog_manager.cc:4841] TS 41c63bb82df844cb94b70282f950bc69 (127.21.2.66:37723): tablet a8348794f6b84f129697bd87eb2a5644 (table test-table [id=54aaa19c95ef4ef3af0b5752b24dcec8]) successfully deleted
I20260927 07:08:21.639670 6061 catalog_manager.cc:4841] TS f39c8cbf952348ad86399d29e5d8d13e (127.21.2.67:41291): tablet a8348794f6b84f129697bd87eb2a5644 (table test-table [id=54aaa19c95ef4ef3af0b5752b24dcec8]) successfully deleted
W20260927 07:08:21.641314 6061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:37369 (error 108)
I20260927 07:08:21.641901 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.641746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37369 (local address 127.0.0.1:40446)
0927 07:08:21.641821 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.641869 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:08:21.641948 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6272
W20260927 07:08:21.643963 6061 catalog_manager.cc:4584] TS 20b7f6a079914b06a46d22e3a36cea29 (127.21.2.65:37369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8348794f6b84f129697bd87eb2a5644: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37369: connect: Connection refused (error 111)
I20260927 07:08:21.647509 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6406
I20260927 07:08:21.652215 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6044
2026-09-27T07:08:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1980 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T07:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:21Z Disabled control of system clock
I20260927 07:08:21.680104 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40177
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:21.761214 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:21.761442 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:21.761462 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:21.762553 6582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:21.762589 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:21.762601 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:21.762614 6582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:21.762626 6582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:21.763938 6582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:21.764128 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:21.764379 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:21.768994 6587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.769022 6588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.769013 6590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.769345 6582 server_base.cc:1034] running on GCE node
I20260927 07:08:21.769591 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:21.769867 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:21.771039 6582 hybrid_clock.cc:648] HybridClock initialized: now 1790492901771029 us; error 27 us; skew 500 ppm
I20260927 07:08:21.772289 6582 webserver.cc:460] Webserver started at http://127.21.2.126:44543/ using document root <none> and password file <none>
I20260927 07:08:21.772544 6582 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:21.772614 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:21.772756 6582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:21.773823 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "cc412defdf414b499f44d598d69bb8cf"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "176561a6d16f58d7cfe75691ac4a2cf2"
server_key_iv: "7e8ea6470fde02035b4b144fb631b2e7"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.774170 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "cc412defdf414b499f44d598d69bb8cf"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "176561a6d16f58d7cfe75691ac4a2cf2"
server_key_iv: "7e8ea6470fde02035b4b144fb631b2e7"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.775529 6582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:21.776388 6596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.776592 6582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:21.776695 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "cc412defdf414b499f44d598d69bb8cf"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "176561a6d16f58d7cfe75691ac4a2cf2"
server_key_iv: "7e8ea6470fde02035b4b144fb631b2e7"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.776770 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:21.792905 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:21.793234 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:21.793401 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:21.797374 6582 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40177
I20260927 07:08:21.797407 6648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40177 every 8 connection(s)
I20260927 07:08:21.797744 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:21.798668 6649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:21.800925 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: Bootstrap starting.
I20260927 07:08:21.801445 6649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:21.801678 6649 log.cc:824] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:21.802284 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: No bootstrap required, opened a new log
I20260927 07:08:21.803906 6649 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } }
I20260927 07:08:21.804041 6649 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:21.804064 6649 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc412defdf414b499f44d598d69bb8cf, State: Initialized, Role: FOLLOWER
I20260927 07:08:21.804219 6649 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } }
I20260927 07:08:21.804291 6649 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:21.804327 6649 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:21.804355 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6582
I20260927 07:08:21.804414 6649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:21.804443 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:21.804644 21513 external_mini_cluster.cc:1428] Setting key 3d4f4b8cfb4572fde5cd7cbb866006d8
I20260927 07:08:21.805083 6649 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } }
I20260927 07:08:21.805194 6649 leader_election.cc:302] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc412defdf414b499f44d598d69bb8cf; no voters:
I20260927 07:08:21.805374 6649 leader_election.cc:288] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:21.805395 6654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:21.805521 6654 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [term 1 LEADER]: Becoming Leader. State: Replica: cc412defdf414b499f44d598d69bb8cf, State: Running, Role: LEADER
I20260927 07:08:21.805605 6654 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } }
I20260927 07:08:21.805634 6649 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:21.805968 6654 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc412defdf414b499f44d598d69bb8cf. Latest consensus state: current_term: 1 leader_uuid: "cc412defdf414b499f44d598d69bb8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } } }
I20260927 07:08:21.806049 6654 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:21.806074 6655 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc412defdf414b499f44d598d69bb8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc412defdf414b499f44d598d69bb8cf" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40177 } } }
I20260927 07:08:21.806147 6655 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:21.806365 6658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:21.806838 6658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:21.808712 6658 catalog_manager.cc:1347] Generated new cluster ID: be421dc7380e4774bd3bee005fb9a9d3
I20260927 07:08:21.808766 6658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:21.810856 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.805178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.21.2.126:40177)
0927 07:08:21.805397 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.805407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.805441 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:08:21.805562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.805570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.805640 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.806046 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.806167 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.807598 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.808684 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.809255 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.809667 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.810105 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.810109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.810109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:21.810300 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.810341 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:21.810354 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.810556 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.810584 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.810651 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260927 07:08:21.817298 6658 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:21.818002 6658 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:21.825356 6658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: Generated new TSK 0
I20260927 07:08:21.825560 6658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:21.832656 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--builtin_ntp_servers=127.21.2.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:21.913404 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:21.913625 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:21.913658 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:21.914769 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:21.914830 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:21.914873 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:21.916239 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:21.916493 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:21.916782 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:21.922433 6679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:21.922572 6678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.922756 6673 server_base.cc:1034] running on GCE node
W20260927 07:08:21.922451 6681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:21.923030 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:21.923288 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:21.924458 6673 hybrid_clock.cc:648] HybridClock initialized: now 1790492901924445 us; error 27 us; skew 500 ppm
I20260927 07:08:21.925854 6673 webserver.cc:460] Webserver started at http://127.21.2.65:45777/ using document root <none> and password file <none>
I20260927 07:08:21.926116 6673 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:21.926174 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:21.926297 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:21.927708 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "012d782caca64c8ca1c4f1e92d1274a9"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "b0d70b5d52eb676c261515c9fbb36d50"
server_key_iv: "c2dfc2f522093efd2971847e0ed634b4"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.928105 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "012d782caca64c8ca1c4f1e92d1274a9"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "b0d70b5d52eb676c261515c9fbb36d50"
server_key_iv: "c2dfc2f522093efd2971847e0ed634b4"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.929662 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:21.930586 6687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.930758 6673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:21.930871 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "012d782caca64c8ca1c4f1e92d1274a9"
format_stamp: "Formatted at 2026-09-27 07:08:21 on dist-test-slave-ltqz"
server_key: "b0d70b5d52eb676c261515c9fbb36d50"
server_key_iv: "c2dfc2f522093efd2971847e0ed634b4"
server_key_version: "encryptionkey@0"
I20260927 07:08:21.930941 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:21.948593 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:21.948951 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:21.949115 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:21.949404 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:21.949774 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:21.949812 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.949837 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:21.949888 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:21.955994 6673 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:44377
I20260927 07:08:21.956081 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:44377 every 8 connection(s)
I20260927 07:08:21.956453 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:21.957834 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6673
I20260927 07:08:21.957911 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:21.958096 21513 external_mini_cluster.cc:1428] Setting key 9afd217778c14d460c3f3fe3d199477a
I20260927 07:08:21.959084 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--builtin_ntp_servers=127.21.2.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:08:21.962383 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.957286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47433 (local address 127.21.2.126:40177)
0927 07:08:21.957337 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:21.957339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:21.957617 (+ 278us) server_negotiation.cc:407] Connection header received
0927 07:08:21.958155 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:21.958157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:21.958174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:21.958255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:21.959082 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.959885 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.960854 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.961172 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.961256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.962118 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:21.962119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:21.962120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:21.962128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:21.962142 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:21.962146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:21.962198 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:21.962272 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 07:08:21.962304 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:08:21.962606 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:21.957262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40177 (local address 127.21.2.65:47433)
0927 07:08:21.957460 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:21.957502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:21.958056 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:21.958353 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:21.958358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:21.958369 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:21.958988 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.958993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.959978 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.959979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.960760 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:21.960761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:21.961267 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:21.961268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:21.961385 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:21.961971 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:21.961971 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:21.961999 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:21.962020 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:21.962195 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:21.962201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:21.962204 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:21.962264 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:21.962373 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260927 07:08:21.963145 6801 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40177
I20260927 07:08:21.963227 6801 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:21.963465 6801 heartbeater.cc:511] Master 127.21.2.126:40177 requested a full tablet report, sending...
I20260927 07:08:21.964044 6613 ts_manager.cc:194] Registered new tserver with Master: 012d782caca64c8ca1c4f1e92d1274a9 (127.21.2.65:44377)
I20260927 07:08:21.965042 6613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.65:47433
W20260927 07:08:22.039960 6804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:22.040158 6804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:22.040179 6804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:22.041345 6804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:22.041388 6804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:22.041414 6804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:22.042740 6804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:22.042987 6804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:22.043268 6804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:22.049115 6810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:22.049165 6804 server_base.cc:1034] running on GCE node
W20260927 07:08:22.049095 6812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:22.049081 6809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:22.049562 6804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:22.049832 6804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:22.051016 6804 hybrid_clock.cc:648] HybridClock initialized: now 1790492902050998 us; error 30 us; skew 500 ppm
I20260927 07:08:22.052448 6804 webserver.cc:460] Webserver started at http://127.21.2.66:36855/ using document root <none> and password file <none>
I20260927 07:08:22.052702 6804 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:22.052774 6804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:22.052930 6804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:22.054128 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "902a1644dab3483e99cb3fe7551044e5"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "05c9eae844de97c0b0b5ea103369c22c"
server_key_iv: "e4e73e3fa07f6390f9e2bd35a8576449"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.054507 6804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "902a1644dab3483e99cb3fe7551044e5"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "05c9eae844de97c0b0b5ea103369c22c"
server_key_iv: "e4e73e3fa07f6390f9e2bd35a8576449"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.055912 6804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:22.056619 6818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.056782 6804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:22.056886 6804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "902a1644dab3483e99cb3fe7551044e5"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "05c9eae844de97c0b0b5ea103369c22c"
server_key_iv: "e4e73e3fa07f6390f9e2bd35a8576449"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.056963 6804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:22.069422 6804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:22.069727 6804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:22.069876 6804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:22.070102 6804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:22.070426 6804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:22.070482 6804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.070524 6804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:22.070552 6804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.076673 6804 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:35495
I20260927 07:08:22.076730 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:35495 every 8 connection(s)
I20260927 07:08:22.077045 6804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:22.082036 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.077746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:33937 (local address 127.21.2.126:40177)
0927 07:08:22.077832 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.077835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.078061 (+ 226us) server_negotiation.cc:407] Connection header received
0927 07:08:22.078519 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.078521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.078530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.078580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:22.079280 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.079904 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.080865 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.081044 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.081095 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.081834 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:22.081835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:22.081835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:22.081839 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:22.081852 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:22.081856 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:22.081897 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.081955 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.081977 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:08:22.082228 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.077716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40177 (local address 127.21.2.66:33937)
0927 07:08:22.077940 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.077973 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.078437 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.078630 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.078633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.078641 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:22.079195 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.079200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.079996 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.079997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.080789 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.080791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.081115 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.081115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.081235 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.081701 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:22.081702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:22.081731 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:22.081752 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:22.081897 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:22.081900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:22.081902 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.081946 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.082019 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":122,"threads_started":1}
I20260927 07:08:22.082772 6932 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40177
I20260927 07:08:22.082861 6932 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:22.083055 6932 heartbeater.cc:511] Master 127.21.2.126:40177 requested a full tablet report, sending...
I20260927 07:08:22.083428 6613 ts_manager.cc:194] Registered new tserver with Master: 902a1644dab3483e99cb3fe7551044e5 (127.21.2.66:35495)
I20260927 07:08:22.083792 6613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.66:33937
I20260927 07:08:22.084271 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6804
I20260927 07:08:22.084396 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:22.084587 21513 external_mini_cluster.cc:1428] Setting key 2fe3c0c26ef4bdea9a9fc03a1943e806
I20260927 07:08:22.085669 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--builtin_ntp_servers=127.21.2.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:22.167313 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:22.167568 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:22.167590 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:22.168715 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:22.168752 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:22.168803 6935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:22.170102 6935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:22.170360 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:22.170637 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:22.175381 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:22.175410 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:22.175551 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:22.175665 6935 server_base.cc:1034] running on GCE node
I20260927 07:08:22.175890 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:22.176142 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:22.177301 6935 hybrid_clock.cc:648] HybridClock initialized: now 1790492902177274 us; error 38 us; skew 500 ppm
I20260927 07:08:22.178673 6935 webserver.cc:460] Webserver started at http://127.21.2.67:43955/ using document root <none> and password file <none>
I20260927 07:08:22.178951 6935 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:22.179023 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:22.179183 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:22.180424 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "a15ee369cfd8488283966822c8942bd3"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "8a38f2c584332a2ccf5218f22215f5fb"
server_key_iv: "53cc644c1d141bb28016c7caebd0b249"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.180795 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "a15ee369cfd8488283966822c8942bd3"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "8a38f2c584332a2ccf5218f22215f5fb"
server_key_iv: "53cc644c1d141bb28016c7caebd0b249"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.182062 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:22.182741 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.182916 6935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:22.183018 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "a15ee369cfd8488283966822c8942bd3"
format_stamp: "Formatted at 2026-09-27 07:08:22 on dist-test-slave-ltqz"
server_key: "8a38f2c584332a2ccf5218f22215f5fb"
server_key_iv: "53cc644c1d141bb28016c7caebd0b249"
server_key_version: "encryptionkey@0"
I20260927 07:08:22.183091 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:22.214514 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:22.214860 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:22.215015 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:22.215261 6935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:22.215646 6935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:22.215700 6935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.215735 6935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:22.215776 6935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:22.221567 6935 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:35663
I20260927 07:08:22.221647 7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:35663 every 8 connection(s)
I20260927 07:08:22.222004 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:22.227341 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.222778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:39727 (local address 127.21.2.126:40177)
0927 07:08:22.222845 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.222847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.223087 (+ 240us) server_negotiation.cc:407] Connection header received
0927 07:08:22.223693 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.223694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.223707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.223754 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:22.224446 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.224972 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.225940 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.226147 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.226228 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.227040 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:22.227042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:22.227042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:22.227047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:22.227063 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:22.227067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:22.227134 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.227225 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.227258 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:08:22.227578 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.222762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:40177 (local address 127.21.2.67:39727)
0927 07:08:22.222971 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.223005 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.223580 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.223825 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.223829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.223837 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:22.224340 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.224344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.225073 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.225074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.225845 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.225847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.226236 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.226237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.226375 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.226899 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:22.226899 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:22.226927 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:22.226949 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:22.227134 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:22.227140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:22.227143 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.227228 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.227347 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260927 07:08:22.228067 7063 heartbeater.cc:347] Connected to a master server at 127.21.2.126:40177
I20260927 07:08:22.228159 7063 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:22.228302 7063 heartbeater.cc:511] Master 127.21.2.126:40177 requested a full tablet report, sending...
I20260927 07:08:22.228636 6613 ts_manager.cc:194] Registered new tserver with Master: a15ee369cfd8488283966822c8942bd3 (127.21.2.67:35663)
I20260927 07:08:22.228964 6613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.67:39727
I20260927 07:08:22.230896 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 6935
I20260927 07:08:22.230976 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:22.231120 21513 external_mini_cluster.cc:1428] Setting key a012d8efae190006e57832d8083fdfd1
I20260927 07:08:22.231921 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:22.236110 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.233229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.21.2.126:40177)
0927 07:08:22.233302 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.233304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.233421 (+ 117us) server_negotiation.cc:407] Connection header received
0927 07:08:22.233490 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.233491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.233503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.233552 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:22.233704 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.234199 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.234946 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.235138 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.235220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.235845 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:22.235847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:22.235847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:22.235851 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:22.235885 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:22.235889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:22.235940 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.236025 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.236048 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:08:22.237385 6613 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:22.239135 6613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:22.248482 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44377 (local address 127.0.0.1:41790)
0927 07:08:22.242499 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.242552 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.242623 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.243353 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.243354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.243357 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:22.243449 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.243452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.244519 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.244520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.246961 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.246964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.248180 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.248181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.248268 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.248283 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.248333 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.248387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":96,"thread_start_us":81,"threads_started":1}
I20260927 07:08:22.248595 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:35495 (local address 127.0.0.1:43366)
0927 07:08:22.242893 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.242903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.242949 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.243338 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.243341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.243345 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:22.243435 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.243437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.245052 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.245053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.247078 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.247082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.248347 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.248349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.248405 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.248409 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.248453 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.248497 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260927 07:08:22.248953 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.21.2.66:35495)
0927 07:08:22.243054 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.243059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.243069 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:22.243111 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.243113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.243129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.243523 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:22.243586 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.244963 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.247191 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.248251 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.248348 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.248447 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.248473 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.248509 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260927 07:08:22.249138 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.21.2.65:44377)
0927 07:08:22.243156 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.243161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.243174 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:08:22.243218 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.243220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.243237 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.243545 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:22.243636 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.244424 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.247069 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.248065 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.248873 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.248968 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.248993 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.249033 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":87,"threads_started":1}
I20260927 07:08:22.250047 6735 tablet_service.cc:1467] Processing CreateTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad (DEFAULT_TABLE table=test-table [id=d5915dbda3f84721a0496b1c6cb21aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:22.250152 6866 tablet_service.cc:1467] Processing CreateTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad (DEFAULT_TABLE table=test-table [id=d5915dbda3f84721a0496b1c6cb21aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:22.250404 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82962e1954b84923beec1c2ff0bbe0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:22.250485 6866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82962e1954b84923beec1c2ff0bbe0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:22.251721 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35663 (local address 127.0.0.1:55104)
0927 07:08:22.242690 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.242700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.242738 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.247660 (+ 4922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.247662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.247665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:22.247738 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.247741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.249216 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.249217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.250401 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.250404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.251498 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.251499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.251562 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.251566 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.251619 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.251657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260927 07:08:22.251783 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.242975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.21.2.67:35663)
0927 07:08:22.247473 (+ 4498us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.247478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.247489 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:08:22.247526 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.247528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.247544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.247632 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:22.247905 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.249086 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.250672 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.251403 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.251470 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.251550 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.251632 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.251699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4452,"thread_start_us":85,"threads_started":1}
I20260927 07:08:22.252367 6997 tablet_service.cc:1467] Processing CreateTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad (DEFAULT_TABLE table=test-table [id=d5915dbda3f84721a0496b1c6cb21aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:22.252709 6997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82962e1954b84923beec1c2ff0bbe0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:22.254252 7077 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Bootstrap starting.
I20260927 07:08:22.255033 7077 tablet_bootstrap.cc:654] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:22.255404 7077 log.cc:824] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:22.255581 7078 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: Bootstrap starting.
I20260927 07:08:22.256184 7078 tablet_bootstrap.cc:654] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:22.256458 7077 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: No bootstrap required, opened a new log
I20260927 07:08:22.256559 7077 ts_tablet_manager.cc:1397] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:08:22.256580 7078 log.cc:824] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:22.256772 7080 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Bootstrap starting.
I20260927 07:08:22.257440 7080 tablet_bootstrap.cc:654] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:22.257603 7078 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: No bootstrap required, opened a new log
I20260927 07:08:22.257676 7078 ts_tablet_manager.cc:1397] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:22.257769 7080 log.cc:824] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:22.259955 7080 tablet_bootstrap.cc:492] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: No bootstrap required, opened a new log
I20260927 07:08:22.260025 7080 ts_tablet_manager.cc:1397] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 07:08:22.260532 7078 raft_consensus.cc:348] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.260720 7078 raft_consensus.cc:374] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:22.260774 7078 raft_consensus.cc:729] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012d782caca64c8ca1c4f1e92d1274a9, State: Initialized, Role: FOLLOWER
I20260927 07:08:22.260941 7078 consensus_queue.cc:260] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.261237 7078 ts_tablet_manager.cc:1428] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 07:08:22.261353 6801 heartbeater.cc:503] Master 127.21.2.126:40177 was elected leader, sending a full tablet report...
I20260927 07:08:22.261763 7077 raft_consensus.cc:348] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.261937 7077 raft_consensus.cc:374] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:22.261971 7077 raft_consensus.cc:729] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a1644dab3483e99cb3fe7551044e5, State: Initialized, Role: FOLLOWER
I20260927 07:08:22.262156 7077 consensus_queue.cc:260] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.262456 7077 ts_tablet_manager.cc:1428] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 07:08:22.262594 6932 heartbeater.cc:503] Master 127.21.2.126:40177 was elected leader, sending a full tablet report...
I20260927 07:08:22.262693 7080 raft_consensus.cc:348] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.262852 7080 raft_consensus.cc:374] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:22.262905 7080 raft_consensus.cc:729] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15ee369cfd8488283966822c8942bd3, State: Initialized, Role: FOLLOWER
I20260927 07:08:22.263020 7080 consensus_queue.cc:260] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.263284 7080 ts_tablet_manager.cc:1428] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:22.263348 7063 heartbeater.cc:503] Master 127.21.2.126:40177 was elected leader, sending a full tablet report...
I20260927 07:08:22.291052 7084 raft_consensus.cc:482] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:22.291270 7084 raft_consensus.cc:504] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.291755 7084 leader_election.cc:288] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 012d782caca64c8ca1c4f1e92d1274a9 (127.21.2.65:44377), a15ee369cfd8488283966822c8942bd3 (127.21.2.67:35663)
I20260927 07:08:22.295486 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.291987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35663 (local address 127.21.2.66:44323)
0927 07:08:22.292181 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.292192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.292237 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.292529 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.292530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.292532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:22.292600 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.292601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.293429 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.293430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.294448 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.294451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.295178 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.295179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.295247 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.295252 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.295331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.295403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":38,"thread_start_us":106,"threads_started":1}
I20260927 07:08:22.295775 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.292068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:44323 (local address 127.21.2.67:35663)
0927 07:08:22.292119 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.292122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.292425 (+ 303us) server_negotiation.cc:407] Connection header received
0927 07:08:22.292459 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.292460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.292468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.292525 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:22.292657 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.293355 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.294518 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.295091 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.295178 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.295203 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.295669 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.295709 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:08:22.295974 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.292221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:47493 (local address 127.21.2.65:44377)
0927 07:08:22.292268 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.292270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.292279 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:22.292306 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.292307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.292319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.292384 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:22.292564 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.293323 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.294736 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.295295 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.295358 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.295367 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.295831 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.295884 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:08:22.296101 7017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82962e1954b84923beec1c2ff0bbe0ad" candidate_uuid: "902a1644dab3483e99cb3fe7551044e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15ee369cfd8488283966822c8942bd3" is_pre_election: true
I20260927 07:08:22.296257 7017 raft_consensus.cc:2454] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 902a1644dab3483e99cb3fe7551044e5 in term 0.
I20260927 07:08:22.296478 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.291894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44377 (local address 127.21.2.66:47493)
0927 07:08:22.292162 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.292176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:22.292240 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:22.292389 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:22.292389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:22.292391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:22.292473 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.292475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.293427 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.293428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.294644 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:22.294646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.295435 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.295435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:22.295507 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.295511 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:22.296359 (+ 848us) client_negotiation.cc:240] Negotiation successful
0927 07:08:22.296408 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":3}
I20260927 07:08:22.296479 6821 leader_election.cc:302] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 902a1644dab3483e99cb3fe7551044e5, a15ee369cfd8488283966822c8942bd3; no voters:
I20260927 07:08:22.296787 7084 raft_consensus.cc:2781] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:22.296901 7084 raft_consensus.cc:482] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:22.296943 7084 raft_consensus.cc:3037] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:22.297021 6755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82962e1954b84923beec1c2ff0bbe0ad" candidate_uuid: "902a1644dab3483e99cb3fe7551044e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012d782caca64c8ca1c4f1e92d1274a9" is_pre_election: true
I20260927 07:08:22.297168 6755 raft_consensus.cc:2454] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 902a1644dab3483e99cb3fe7551044e5 in term 0.
I20260927 07:08:22.297946 7084 raft_consensus.cc:504] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.298208 7084 leader_election.cc:288] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [CANDIDATE]: Term 1 election: Requested vote from peers 012d782caca64c8ca1c4f1e92d1274a9 (127.21.2.65:44377), a15ee369cfd8488283966822c8942bd3 (127.21.2.67:35663)
I20260927 07:08:22.298236 6755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82962e1954b84923beec1c2ff0bbe0ad" candidate_uuid: "902a1644dab3483e99cb3fe7551044e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012d782caca64c8ca1c4f1e92d1274a9"
I20260927 07:08:22.298332 6755 raft_consensus.cc:3037] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:22.299307 6755 raft_consensus.cc:2454] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 902a1644dab3483e99cb3fe7551044e5 in term 1.
I20260927 07:08:22.299510 7017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82962e1954b84923beec1c2ff0bbe0ad" candidate_uuid: "902a1644dab3483e99cb3fe7551044e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15ee369cfd8488283966822c8942bd3"
I20260927 07:08:22.299602 7017 raft_consensus.cc:3037] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:22.300397 7017 raft_consensus.cc:2454] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 902a1644dab3483e99cb3fe7551044e5 in term 1.
I20260927 07:08:22.300534 6821 leader_election.cc:302] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012d782caca64c8ca1c4f1e92d1274a9, 902a1644dab3483e99cb3fe7551044e5; no voters:
I20260927 07:08:22.300644 7084 raft_consensus.cc:2781] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:22.300808 7084 raft_consensus.cc:686] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 1 LEADER]: Becoming Leader. State: Replica: 902a1644dab3483e99cb3fe7551044e5, State: Running, Role: LEADER
I20260927 07:08:22.300920 7084 consensus_queue.cc:237] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } }
I20260927 07:08:22.301703 6613 catalog_manager.cc:5495] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 902a1644dab3483e99cb3fe7551044e5 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "902a1644dab3483e99cb3fe7551044e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "902a1644dab3483e99cb3fe7551044e5" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 35495 } health_report { overall_health: HEALTHY } } }
I20260927 07:08:22.307183 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.304688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.21.2.66:35495)
0927 07:08:22.304739 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:22.304741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:22.304750 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:22.304797 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:22.304797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:22.304813 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:22.304886 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:22.304981 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.305503 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.306333 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:22.306583 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:22.306658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:22.306827 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:22.306928 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:22.307043 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:22.307065 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 07:08:22.307084 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 07:08:22.308804 7017 raft_consensus.cc:1261] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 1 FOLLOWER]: Refusing update from remote peer 902a1644dab3483e99cb3fe7551044e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:22.308835 6755 raft_consensus.cc:1261] T 82962e1954b84923beec1c2ff0bbe0ad P 012d782caca64c8ca1c4f1e92d1274a9 [term 1 FOLLOWER]: Refusing update from remote peer 902a1644dab3483e99cb3fe7551044e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:22.309142 7087 consensus_queue.cc:1035] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15ee369cfd8488283966822c8942bd3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:22.309294 7084 consensus_queue.cc:1035] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012d782caca64c8ca1c4f1e92d1274a9" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:22.312332 7091 mvcc.cc:205] Tried to move back new op lower bound from 7333858927854489600 to 7333858927825244160. Current Snapshot: MvccSnapshot[applied={T|T < 7333858927854489600}]
I20260927 07:08:22.313853 6613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:22.313938 6613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:22.314652 6613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc412defdf414b499f44d598d69bb8cf: Sending DeleteTablet for 3 replicas of tablet 82962e1954b84923beec1c2ff0bbe0ad
I20260927 07:08:22.314968 6735 tablet_service.cc:1514] Processing DeleteTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:22 UTC) from {username='slave'} at 127.0.0.1:41790
I20260927 07:08:22.315042 6997 tablet_service.cc:1514] Processing DeleteTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:22 UTC) from {username='slave'} at 127.0.0.1:55104
I20260927 07:08:22.315084 6866 tablet_service.cc:1514] Processing DeleteTablet for tablet 82962e1954b84923beec1c2ff0bbe0ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:22 UTC) from {username='slave'} at 127.0.0.1:43366
I20260927 07:08:22.315292 7102 tablet_replica.cc:330] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: stopping tablet replica
I20260927 07:08:22.315387 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6673
I20260927 07:08:22.315412 7102 raft_consensus.cc:2229] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:22.316303 7102 raft_consensus.cc:2258] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:22.316481 7101 tablet_replica.cc:330] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: stopping tablet replica
I20260927 07:08:22.316576 7101 raft_consensus.cc:2229] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:22.316680 7101 raft_consensus.cc:2258] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:22.316865 7102 ts_tablet_manager.cc:1905] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:22.317085 7101 ts_tablet_manager.cc:1905] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:22.318580 7102 ts_tablet_manager.cc:1918] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:22.318737 7102 log.cc:1196] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/82962e1954b84923beec1c2ff0bbe0ad
I20260927 07:08:22.318885 7101 ts_tablet_manager.cc:1918] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:22.318974 7101 log.cc:1196] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/82962e1954b84923beec1c2ff0bbe0ad
I20260927 07:08:22.319168 7102 ts_tablet_manager.cc:1939] T 82962e1954b84923beec1c2ff0bbe0ad P 902a1644dab3483e99cb3fe7551044e5: Deleting consensus metadata
I20260927 07:08:22.319211 7101 ts_tablet_manager.cc:1939] T 82962e1954b84923beec1c2ff0bbe0ad P a15ee369cfd8488283966822c8942bd3: Deleting consensus metadata
I20260927 07:08:22.319687 6599 catalog_manager.cc:4841] TS 902a1644dab3483e99cb3fe7551044e5 (127.21.2.66:35495): tablet 82962e1954b84923beec1c2ff0bbe0ad (table test-table [id=d5915dbda3f84721a0496b1c6cb21aff]) successfully deleted
I20260927 07:08:22.319782 6599 catalog_manager.cc:4841] TS a15ee369cfd8488283966822c8942bd3 (127.21.2.67:35663): tablet 82962e1954b84923beec1c2ff0bbe0ad (table test-table [id=d5915dbda3f84721a0496b1c6cb21aff]) successfully deleted
W20260927 07:08:22.322108 6599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:44377 (error 108)
I20260927 07:08:22.322618 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:22.322473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:44377 (local address 127.0.0.1:41802)
0927 07:08:22.322534 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:22.322579 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:44377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:08:22.322700 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6804
W20260927 07:08:22.324049 6599 catalog_manager.cc:4584] TS 012d782caca64c8ca1c4f1e92d1274a9 (127.21.2.65:44377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82962e1954b84923beec1c2ff0bbe0ad: Network error: Client connection negotiation failed: client connection to 127.21.2.65:44377: connect: Connection refused (error 111)
I20260927 07:08:22.327524 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6935
I20260927 07:08:22.332075 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 6582
2026-09-27T07:08:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (678 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:22 dist-test-slave-ltqz krb5kdc[7106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:22 dist-test-slave-ltqz krb5kdc[7106](info): set up 2 sockets
Sep 27 07:08:22 dist-test-slave-ltqz krb5kdc[7106](info): commencing operation
krb5kdc: starting...
W20260927 07:08:23.391198 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:23.507893 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42497
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:42497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:23.589056 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:23.589309 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:23.589349 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:23.590516 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:23.590584 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:23.590610 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:23.590638 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:23.590667 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:23.592231 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:42497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:23.592504 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:23.592728 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:23.597420 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.597473 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.597425 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:23.597596 7119 server_base.cc:1034] running on GCE node
I20260927 07:08:23.597990 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:23.598269 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:23.599440 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790492903599431 us; error 33 us; skew 500 ppm
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:23.601799 7119 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:23.602277 7119 webserver.cc:460] Webserver started at http://127.21.2.126:39563/ using document root <none> and password file <none>
I20260927 07:08:23.602502 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:23.602571 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:23.602722 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:23.603834 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "e184372a4b4840db978bd5860535bbad"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "a2cb1d0432e7b977c065f48af0cbc272"
server_key_iv: "4141986d908b229d0bff72bfc22ab83f"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.604210 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "e184372a4b4840db978bd5860535bbad"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "a2cb1d0432e7b977c065f48af0cbc272"
server_key_iv: "4141986d908b229d0bff72bfc22ab83f"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.605535 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:23.606283 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.606449 7119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:23.606554 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "e184372a4b4840db978bd5860535bbad"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "a2cb1d0432e7b977c065f48af0cbc272"
server_key_iv: "4141986d908b229d0bff72bfc22ab83f"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.606627 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:23.627063 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:23.628960 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:23.629120 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:23.633244 7119 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:42497
I20260927 07:08:23.633293 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:42497 every 8 connection(s)
I20260927 07:08:23.633646 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:23.634502 7187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:23.636735 7187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: Bootstrap starting.
I20260927 07:08:23.637315 7187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:23.637609 7187 log.cc:824] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:23.638329 7187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: No bootstrap required, opened a new log
I20260927 07:08:23.640002 7187 raft_consensus.cc:348] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } }
I20260927 07:08:23.640141 7187 raft_consensus.cc:374] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:23.640194 7187 raft_consensus.cc:729] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e184372a4b4840db978bd5860535bbad, State: Initialized, Role: FOLLOWER
I20260927 07:08:23.640379 7187 consensus_queue.cc:260] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } }
I20260927 07:08:23.640466 7187 raft_consensus.cc:388] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:23.640517 7187 raft_consensus.cc:482] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:23.640568 7187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:23.641099 7187 raft_consensus.cc:504] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } }
I20260927 07:08:23.641214 7187 leader_election.cc:302] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e184372a4b4840db978bd5860535bbad; no voters:
I20260927 07:08:23.641386 7187 leader_election.cc:288] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:23.641448 7190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:23.641620 7190 raft_consensus.cc:686] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [term 1 LEADER]: Becoming Leader. State: Replica: e184372a4b4840db978bd5860535bbad, State: Running, Role: LEADER
I20260927 07:08:23.641629 7187 sys_catalog.cc:564] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:23.641727 7190 consensus_queue.cc:237] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } }
I20260927 07:08:23.642143 7192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e184372a4b4840db978bd5860535bbad. Latest consensus state: current_term: 1 leader_uuid: "e184372a4b4840db978bd5860535bbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } } }
I20260927 07:08:23.642230 7192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:23.642373 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e184372a4b4840db978bd5860535bbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e184372a4b4840db978bd5860535bbad" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42497 } } }
I20260927 07:08:23.642464 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:23.642701 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7119
I20260927 07:08:23.642776 7199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:23.642834 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:23.643016 21513 external_mini_cluster.cc:1428] Setting key 88e1372e18cd935dea4fdea0dae1e858
I20260927 07:08:23.643256 7199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:23.646014 7199 catalog_manager.cc:1347] Generated new cluster ID: bff2ce28d319486b928808df771f4ddb
I20260927 07:08:23.646080 7199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:23.655577 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:23.643530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.21.2.126:42497)
0927 07:08:23.643821 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:23.643830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:23.643860 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:08:23.643977 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:23.643984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:23.644126 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:23.644271 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:23.645417 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.646966 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.648091 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.648732 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.648851 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:23.651820 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:23.651825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:23.651825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:23.651837 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:23.653878 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.654280 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.654283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.654284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.654408 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.654610 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.654611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.654612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.654917 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:23.655082 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:23.655157 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 07:08:23.655310 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":67,"threads_started":1}
I20260927 07:08:23.659941 7199 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:23.660485 7199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:23.664315 7199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: Generated new TSK 0
I20260927 07:08:23.664489 7199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:08:23.694335 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:23.776391 7213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:23.776602 7213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:23.776626 7213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:23.777813 7213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:23.777855 7213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:23.777885 7213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:23.779332 7213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:23.779600 7213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:23.779840 7213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:23.785120 7213 server_base.cc:1034] running on GCE node
W20260927 07:08:23.785082 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.785089 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.785089 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:23.785535 7213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:23.785778 7213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:23.786938 7213 hybrid_clock.cc:648] HybridClock initialized: now 1790492903786934 us; error 35 us; skew 500 ppm
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:23.789459 7213 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:08:23.789904 7213 webserver.cc:460] Webserver started at http://127.21.2.65:39099/ using document root <none> and password file <none>
I20260927 07:08:23.790143 7213 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:23.790210 7213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:23.790362 7213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:23.791456 7213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "dc4fdcddff7942228a37f6b926c7650d"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "2e590bcb66213fe921db0dd7cf089bed"
server_key_iv: "02913c2a10b182c90a3e54c83fe87e05"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.791834 7213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "dc4fdcddff7942228a37f6b926c7650d"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "2e590bcb66213fe921db0dd7cf089bed"
server_key_iv: "02913c2a10b182c90a3e54c83fe87e05"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.793315 7213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:08:23.794238 7228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.794436 7213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:23.794539 7213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "dc4fdcddff7942228a37f6b926c7650d"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "2e590bcb66213fe921db0dd7cf089bed"
server_key_iv: "02913c2a10b182c90a3e54c83fe87e05"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.794636 7213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:23.807349 7213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:23.809322 7213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:23.809480 7213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:23.809733 7213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:23.810058 7213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:23.810119 7213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.810164 7213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:23.810191 7213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.815884 7213 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:40015
I20260927 07:08:23.815963 7341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:40015 every 8 connection(s)
I20260927 07:08:23.816286 7213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:23.819749 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7213
I20260927 07:08:23.819873 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:23.820073 21513 external_mini_cluster.cc:1428] Setting key 047321e14c0b15c30bf127fde522b1c7
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:23.826562 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:23.816955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42497 (local address 127.21.2.65:37353)
0927 07:08:23.817131 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:23.817164 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:23.817653 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:23.817904 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:23.817909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:23.818470 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:23.818650 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:23.818654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.819355 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.819357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:23.820618 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:23.820622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.821058 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.821059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:23.821214 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:23.821756 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:23.821767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:23.823294 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:23.825233 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:23.825240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:23.825241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:23.825446 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:23.825686 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:23.825686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:23.825687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:23.825786 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:23.826085 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:23.826090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:23.826202 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 07:08:23.826283 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:08:23.826401 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260927 07:08:23.826584 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:23.816967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:37353 (local address 127.21.2.126:42497)
0927 07:08:23.817010 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:23.817012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:23.817244 (+ 232us) server_negotiation.cc:407] Connection header received
0927 07:08:23.817763 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:23.817764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:23.817779 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:23.817838 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:23.818748 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.819256 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.820705 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.820968 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.821030 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:23.823459 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:23.823461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:23.823462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:23.823469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:23.825137 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.825526 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.825528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.825528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.825610 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.825854 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.825856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.825856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.826014 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:23.826095 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:23.826336 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 07:08:23.826433 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:08:23.827090 7342 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42497
I20260927 07:08:23.827196 7342 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:23.827423 7342 heartbeater.cc:511] Master 127.21.2.126:42497 requested a full tablet report, sending...
I20260927 07:08:23.827970 7151 ts_manager.cc:194] Registered new tserver with Master: dc4fdcddff7942228a37f6b926c7650d (127.21.2.65:40015)
I20260927 07:08:23.828585 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:37353
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:08:23.848210 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:23.929452 7347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:23.929661 7347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:23.929684 7347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:23.930866 7347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:23.930913 7347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:23.930940 7347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:23.932505 7347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:23.932783 7347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:23.933019 7347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:23.937909 7355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.937968 7353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:23.937918 7352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:23.938064 7347 server_base.cc:1034] running on GCE node
I20260927 07:08:23.938305 7347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:23.938586 7347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:23.939749 7347 hybrid_clock.cc:648] HybridClock initialized: now 1790492903939735 us; error 29 us; skew 500 ppm
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:23.942155 7347 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:08:23.942596 7347 webserver.cc:460] Webserver started at http://127.21.2.66:42865/ using document root <none> and password file <none>
I20260927 07:08:23.942876 7347 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:23.942948 7347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:23.943109 7347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:23.944259 7347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "90da36a179ce42e38a3a1318ae08534c"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "d6ab48eba045878dc6d2a881494fd440"
server_key_iv: "2d7daba253da0eb704b04fc3bf40a05a"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.944630 7347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "90da36a179ce42e38a3a1318ae08534c"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "d6ab48eba045878dc6d2a881494fd440"
server_key_iv: "2d7daba253da0eb704b04fc3bf40a05a"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.947237 7347 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:08:23.948242 7362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.948441 7347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:23.948547 7347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "90da36a179ce42e38a3a1318ae08534c"
format_stamp: "Formatted at 2026-09-27 07:08:23 on dist-test-slave-ltqz"
server_key: "d6ab48eba045878dc6d2a881494fd440"
server_key_iv: "2d7daba253da0eb704b04fc3bf40a05a"
server_key_version: "encryptionkey@0"
I20260927 07:08:23.948639 7347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:23.959906 7347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:23.962234 7347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:23.962352 7347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:23.962535 7347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:23.962864 7347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:23.962922 7347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.962966 7347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:23.962992 7347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:23.968998 7347 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:36021
I20260927 07:08:23.969079 7475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:36021 every 8 connection(s)
I20260927 07:08:23.969451 7347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:23.973433 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7347
I20260927 07:08:23.973569 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:23.973750 21513 external_mini_cluster.cc:1428] Setting key fc8162c18a6fada7ecf882ab6365fe6a
Sep 27 07:08:23 dist-test-slave-ltqz krb5kdc[7106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492903, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:23.979874 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:23.970313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:44549 (local address 127.21.2.126:42497)
0927 07:08:23.970381 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:23.970384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:23.970631 (+ 247us) server_negotiation.cc:407] Connection header received
0927 07:08:23.971218 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:23.971219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:23.971234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:23.971306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:23.972331 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.972860 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.974094 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.974389 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.974462 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:23.976906 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:23.976909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:23.976910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:23.976918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:23.978422 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.978858 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.978860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.978860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.978923 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:23.979190 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:23.979191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:23.979192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:23.979314 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:23.979405 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:23.979697 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 07:08:23.979784 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:23.980080 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:23.970259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42497 (local address 127.21.2.66:44549)
0927 07:08:23.970473 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:23.970523 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:23.971091 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:23.971409 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:23.971413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:23.972011 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:23.972216 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:23.972221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.972982 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.972983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:23.973974 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:23.973977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:23.974506 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:23.974508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:23.974663 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:23.975235 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:23.975244 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:23.976790 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:23.978537 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:23.978545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:23.978547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:23.978776 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:23.978993 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:23.978994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:23.978994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:23.979116 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:23.979411 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:23.979419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:23.979557 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:08:23.979664 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 07:08:23.979829 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260927 07:08:23.980679 7476 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42497
I20260927 07:08:23.980787 7476 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:23.980968 7476 heartbeater.cc:511] Master 127.21.2.126:42497 requested a full tablet report, sending...
I20260927 07:08:23.981442 7151 ts_manager.cc:194] Registered new tserver with Master: 90da36a179ce42e38a3a1318ae08534c (127.21.2.66:36021)
I20260927 07:08:23.981976 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:44549
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:08:24.000134 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:24.082978 7481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:24.083191 7481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:24.083223 7481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:24.084445 7481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:24.084489 7481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:24.084517 7481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:24.085974 7481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:24.086172 7481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:24.086369 7481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:24.091352 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.091331 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.091472 7481 server_base.cc:1034] running on GCE node
W20260927 07:08:24.091601 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.091814 7481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:24.092077 7481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:24.093264 7481 hybrid_clock.cc:648] HybridClock initialized: now 1790492904093251 us; error 39 us; skew 500 ppm
Sep 27 07:08:24 dist-test-slave-ltqz krb5kdc[7106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492904, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:24.095922 7481 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:08:24.096423 7481 webserver.cc:460] Webserver started at http://127.21.2.67:34039/ using document root <none> and password file <none>
I20260927 07:08:24.096668 7481 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:24.096736 7481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:24.096886 7481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:24.098008 7481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "306242ed45b54edeb91e77a28eb710c3"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "c30ecaca2f16e894ea7f91c2895d5739"
server_key_iv: "a4136105a06635fa8c76ad34defeb906"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.098387 7481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "306242ed45b54edeb91e77a28eb710c3"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "c30ecaca2f16e894ea7f91c2895d5739"
server_key_iv: "a4136105a06635fa8c76ad34defeb906"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.099787 7481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:24.100520 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.100696 7481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:24.100801 7481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "306242ed45b54edeb91e77a28eb710c3"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "c30ecaca2f16e894ea7f91c2895d5739"
server_key_iv: "a4136105a06635fa8c76ad34defeb906"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.100879 7481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:24.114207 7481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:24.116529 7481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:24.116690 7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:24.116914 7481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:24.117218 7481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:24.117272 7481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.117314 7481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:24.117342 7481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.122975 7481 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:35409
I20260927 07:08:24.123024 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:35409 every 8 connection(s)
I20260927 07:08:24.123342 7481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:24.125174 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7481
I20260927 07:08:24.125298 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:24.125509 21513 external_mini_cluster.cc:1428] Setting key e924e0e0053cc2bec055bbe8a3777d13
Sep 27 07:08:24 dist-test-slave-ltqz krb5kdc[7106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492904, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:24.132836 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.124094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:54719 (local address 127.21.2.126:42497)
0927 07:08:24.124137 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.124139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.124338 (+ 199us) server_negotiation.cc:407] Connection header received
0927 07:08:24.125046 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.125048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.125061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.125160 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:24.126184 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.126677 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.127764 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.127959 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.128044 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.130147 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:24.130149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:24.130149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:24.130155 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:24.131720 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:24.132060 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:24.132061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:24.132061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:24.132128 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:24.132338 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:24.132339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:24.132339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:24.132438 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:24.132492 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.132711 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.132769 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:24.133006 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.124053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42497 (local address 127.21.2.67:54719)
0927 07:08:24.124201 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.124251 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.124910 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.125244 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.125248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.125895 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:24.126086 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.126090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.126786 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.126787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.127669 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.127672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.128054 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.128055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.128183 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.128689 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:24.128696 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:24.130018 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:24.131789 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:24.131794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:24.131796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:24.131997 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:24.132185 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:24.132185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:24.132185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:24.132285 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:24.132496 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:24.132500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:24.132609 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.132688 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.132805 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260927 07:08:24.133481 7610 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42497
I20260927 07:08:24.133587 7610 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:24.133769 7610 heartbeater.cc:511] Master 127.21.2.126:42497 requested a full tablet report, sending...
I20260927 07:08:24.134163 7151 ts_manager.cc:194] Registered new tserver with Master: 306242ed45b54edeb91e77a28eb710c3 (127.21.2.67:35409)
I20260927 07:08:24.134644 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:54719
I20260927 07:08:24.137080 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:24.145749 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.138444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.21.2.126:42497)
0927 07:08:24.138518 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.138520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.138598 (+ 78us) server_negotiation.cc:407] Connection header received
0927 07:08:24.138633 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.138634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.138642 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.138688 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:24.139197 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.139733 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.140449 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.140640 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.140704 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.142045 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:24.142046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:24.142047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:24.142054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:24.143625 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:24.143996 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:24.143997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:24.143997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:24.144077 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:24.144437 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:24.144438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:24.144438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:24.144589 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:24.144708 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.145586 (+ 878us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.145668 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:08:24.148067 7151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:24.149652 7151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:24.156334 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.151955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:40015 (local address 127.0.0.1:48322)
0927 07:08:24.152262 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.152301 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.152371 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.152575 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.152578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.152580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:24.152652 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.152655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.153980 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.153981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.155121 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.155124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.156036 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.156037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.156120 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156123 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.156180 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.156226 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260927 07:08:24.156342 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.152225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.21.2.65:40015)
0927 07:08:24.152414 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.152419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.152429 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:24.152469 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.152471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.152485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.152554 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:24.152740 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.153879 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155194 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.155881 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155944 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.156205 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.156262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260927 07:08:24.156404 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.152208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.21.2.67:35409)
0927 07:08:24.152423 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.152427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.152578 (+ 151us) server_negotiation.cc:407] Connection header received
0927 07:08:24.152618 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.152621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.152637 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.152730 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:24.152994 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.154034 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155116 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.155862 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155944 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.156275 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.156317 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260927 07:08:24.156467 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.152044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.21.2.66:36021)
0927 07:08:24.152660 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.152664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.152671 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:08:24.152726 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.152728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.152742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.152816 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:24.153000 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.153708 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155436 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.155991 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.156062 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.156327 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.156380 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":113,"threads_started":1}
I20260927 07:08:24.156546 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.151946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35409 (local address 127.0.0.1:37536)
0927 07:08:24.152442 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.152456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.152513 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.152798 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.152800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.152802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:24.152850 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.152852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.154144 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.154145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.155027 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.155029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.155956 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.155957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.156042 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156058 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.156125 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.156186 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260927 07:08:24.156623 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.151946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:36021 (local address 127.0.0.1:33028)
0927 07:08:24.152256 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.152301 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.152370 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.152895 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.152896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.152898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:24.152929 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.152930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.153808 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.153809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.155121 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.155124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.156061 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.156062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.156117 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.156121 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.156185 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.156227 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":46,"thread_start_us":60,"threads_started":1}
I20260927 07:08:24.157333 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 24c7cdad29e24e408fb150dc07c65432 (DEFAULT_TABLE table=test-table [id=56b822dcfe3f4ee4a22128a0adcb90ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:24.157699 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c7cdad29e24e408fb150dc07c65432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:24.157650 7410 tablet_service.cc:1467] Processing CreateTablet for tablet 24c7cdad29e24e408fb150dc07c65432 (DEFAULT_TABLE table=test-table [id=56b822dcfe3f4ee4a22128a0adcb90ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:24.157701 7276 tablet_service.cc:1467] Processing CreateTablet for tablet 24c7cdad29e24e408fb150dc07c65432 (DEFAULT_TABLE table=test-table [id=56b822dcfe3f4ee4a22128a0adcb90ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:24.157943 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c7cdad29e24e408fb150dc07c65432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:24.157994 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c7cdad29e24e408fb150dc07c65432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:24.160511 7624 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Bootstrap starting.
I20260927 07:08:24.161048 7626 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Bootstrap starting.
I20260927 07:08:24.161283 7624 tablet_bootstrap.cc:654] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:24.161736 7626 tablet_bootstrap.cc:654] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:24.161877 7624 log.cc:824] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:24.162264 7626 log.cc:824] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:24.162613 7625 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Bootstrap starting.
I20260927 07:08:24.162662 7624 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: No bootstrap required, opened a new log
I20260927 07:08:24.162715 7624 ts_tablet_manager.cc:1397] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:08:24.163224 7625 tablet_bootstrap.cc:654] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:24.163743 7626 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: No bootstrap required, opened a new log
I20260927 07:08:24.163791 7625 log.cc:824] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:24.163815 7626 ts_tablet_manager.cc:1397] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:08:24.164791 7624 raft_consensus.cc:348] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.164954 7624 raft_consensus.cc:374] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:24.164978 7624 raft_consensus.cc:729] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90da36a179ce42e38a3a1318ae08534c, State: Initialized, Role: FOLLOWER
I20260927 07:08:24.165091 7624 consensus_queue.cc:260] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.165387 7624 ts_tablet_manager.cc:1428] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:24.165378 7626 raft_consensus.cc:348] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.165531 7476 heartbeater.cc:503] Master 127.21.2.126:42497 was elected leader, sending a full tablet report...
I20260927 07:08:24.165555 7626 raft_consensus.cc:374] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:24.165592 7626 raft_consensus.cc:729] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4fdcddff7942228a37f6b926c7650d, State: Initialized, Role: FOLLOWER
I20260927 07:08:24.165704 7626 consensus_queue.cc:260] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.165973 7626 ts_tablet_manager.cc:1428] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:08:24.166131 7342 heartbeater.cc:503] Master 127.21.2.126:42497 was elected leader, sending a full tablet report...
I20260927 07:08:24.166659 7625 tablet_bootstrap.cc:492] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: No bootstrap required, opened a new log
I20260927 07:08:24.166725 7625 ts_tablet_manager.cc:1397] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 07:08:24.168473 7625 raft_consensus.cc:348] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.168601 7625 raft_consensus.cc:374] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:24.168622 7625 raft_consensus.cc:729] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306242ed45b54edeb91e77a28eb710c3, State: Initialized, Role: FOLLOWER
I20260927 07:08:24.168761 7625 consensus_queue.cc:260] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.169052 7625 ts_tablet_manager.cc:1428] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 07:08:24.169096 7610 heartbeater.cc:503] Master 127.21.2.126:42497 was elected leader, sending a full tablet report...
W20260927 07:08:24.220171 7477 tablet.cc:2369] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:08:24.317130 7343 tablet.cc:2369] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:24.363732 7631 raft_consensus.cc:482] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:24.364017 7631 raft_consensus.cc:504] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.364441 7631 leader_election.cc:288] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 306242ed45b54edeb91e77a28eb710c3 (127.21.2.67:35409), 90da36a179ce42e38a3a1318ae08534c (127.21.2.66:36021)
I20260927 07:08:24.368716 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.364742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:51189 (local address 127.21.2.67:35409)
0927 07:08:24.364815 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.364817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.364886 (+ 69us) server_negotiation.cc:407] Connection header received
0927 07:08:24.364982 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.364983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.364998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.365075 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:24.365317 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.366069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.367718 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.368323 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.368409 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.368422 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.368594 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.368638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 07:08:24.368716 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.364741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54989 (local address 127.21.2.66:36021)
0927 07:08:24.364812 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.364815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.365052 (+ 237us) server_negotiation.cc:407] Connection header received
0927 07:08:24.365084 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.365085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.365098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.365157 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:24.365317 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.366085 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.367717 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.368320 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.368409 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.368422 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.368595 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.368638 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:24.368755 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.364642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:35409 (local address 127.21.2.65:51189)
0927 07:08:24.364799 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.364811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.364905 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.365090 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.365091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.365107 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:24.365219 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.365222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.366171 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.366172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.367612 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.367616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.368422 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.368423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.368506 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.368513 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.368573 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.368629 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":105,"thread_start_us":74,"threads_started":1}
I20260927 07:08:24.368763 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.364642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:36021 (local address 127.21.2.65:54989)
0927 07:08:24.364900 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.364909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.364961 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.365171 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.365172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.365174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:24.365242 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.365243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.366171 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.366172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.367612 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.367616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.368420 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.368421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.368506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.368513 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.368572 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.368629 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":88,"thread_start_us":71,"threads_started":1}
I20260927 07:08:24.369190 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c7cdad29e24e408fb150dc07c65432" candidate_uuid: "dc4fdcddff7942228a37f6b926c7650d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90da36a179ce42e38a3a1318ae08534c" is_pre_election: true
I20260927 07:08:24.369194 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c7cdad29e24e408fb150dc07c65432" candidate_uuid: "dc4fdcddff7942228a37f6b926c7650d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306242ed45b54edeb91e77a28eb710c3" is_pre_election: true
I20260927 07:08:24.369360 7564 raft_consensus.cc:2454] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4fdcddff7942228a37f6b926c7650d in term 0.
I20260927 07:08:24.369360 7430 raft_consensus.cc:2454] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4fdcddff7942228a37f6b926c7650d in term 0.
I20260927 07:08:24.369601 7231 leader_election.cc:302] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306242ed45b54edeb91e77a28eb710c3, dc4fdcddff7942228a37f6b926c7650d; no voters:
I20260927 07:08:24.369731 7631 raft_consensus.cc:2781] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:24.369786 7631 raft_consensus.cc:482] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:24.369809 7631 raft_consensus.cc:3037] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:24.370535 7631 raft_consensus.cc:504] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.370769 7631 leader_election.cc:288] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [CANDIDATE]: Term 1 election: Requested vote from peers 306242ed45b54edeb91e77a28eb710c3 (127.21.2.67:35409), 90da36a179ce42e38a3a1318ae08534c (127.21.2.66:36021)
I20260927 07:08:24.370867 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c7cdad29e24e408fb150dc07c65432" candidate_uuid: "dc4fdcddff7942228a37f6b926c7650d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90da36a179ce42e38a3a1318ae08534c"
I20260927 07:08:24.370923 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c7cdad29e24e408fb150dc07c65432" candidate_uuid: "dc4fdcddff7942228a37f6b926c7650d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306242ed45b54edeb91e77a28eb710c3"
I20260927 07:08:24.370946 7430 raft_consensus.cc:3037] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:24.370988 7564 raft_consensus.cc:3037] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:24.371863 7564 raft_consensus.cc:2454] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4fdcddff7942228a37f6b926c7650d in term 1.
I20260927 07:08:24.371863 7430 raft_consensus.cc:2454] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4fdcddff7942228a37f6b926c7650d in term 1.
I20260927 07:08:24.372092 7232 leader_election.cc:302] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90da36a179ce42e38a3a1318ae08534c, dc4fdcddff7942228a37f6b926c7650d; no voters:
I20260927 07:08:24.372196 7631 raft_consensus.cc:2781] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:24.372372 7631 raft_consensus.cc:686] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 1 LEADER]: Becoming Leader. State: Replica: dc4fdcddff7942228a37f6b926c7650d, State: Running, Role: LEADER
I20260927 07:08:24.372488 7631 consensus_queue.cc:237] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } }
I20260927 07:08:24.373308 7151 catalog_manager.cc:5495] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d reported cstate change: term changed from 0 to 1, leader changed from <none> to dc4fdcddff7942228a37f6b926c7650d (127.21.2.65). New cstate: current_term: 1 leader_uuid: "dc4fdcddff7942228a37f6b926c7650d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc4fdcddff7942228a37f6b926c7650d" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 40015 } health_report { overall_health: HEALTHY } } }
W20260927 07:08:24.373879 7611 tablet.cc:2369] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:24.386132 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.383775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.21.2.65:40015)
0927 07:08:24.383848 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.383850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.383861 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:08:24.383907 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.383908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.383927 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.384004 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:24.384139 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.384594 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.385363 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.385572 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.385658 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.385801 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:24.385897 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:24.386008 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.386046 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.386054 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:08:24.387888 7564 raft_consensus.cc:1261] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 1 FOLLOWER]: Refusing update from remote peer dc4fdcddff7942228a37f6b926c7650d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:24.387913 7430 raft_consensus.cc:1261] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 1 FOLLOWER]: Refusing update from remote peer dc4fdcddff7942228a37f6b926c7650d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:24.388247 7631 consensus_queue.cc:1035] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [LEADER]: Connected to new peer: Peer: permanent_uuid: "306242ed45b54edeb91e77a28eb710c3" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:24.388399 7635 consensus_queue.cc:1035] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90da36a179ce42e38a3a1318ae08534c" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 36021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:24.390739 7639 mvcc.cc:205] Tried to move back new op lower bound from 7333858936369913856 to 7333858936310411264. Current Snapshot: MvccSnapshot[applied={T|T < 7333858936369913856}]
I20260927 07:08:24.393319 7150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:24.393422 7150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:24.394261 7150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e184372a4b4840db978bd5860535bbad: Sending DeleteTablet for 3 replicas of tablet 24c7cdad29e24e408fb150dc07c65432
I20260927 07:08:24.394651 7410 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c7cdad29e24e408fb150dc07c65432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:33028
I20260927 07:08:24.394644 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c7cdad29e24e408fb150dc07c65432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:37536
I20260927 07:08:24.394917 7648 tablet_replica.cc:330] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: stopping tablet replica
I20260927 07:08:24.395023 7648 raft_consensus.cc:2229] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:24.395004 7276 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c7cdad29e24e408fb150dc07c65432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:24 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:48322
I20260927 07:08:24.395128 7648 raft_consensus.cc:2258] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:24.395284 7649 tablet_replica.cc:330] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: stopping tablet replica
I20260927 07:08:24.395344 7650 tablet_replica.cc:330] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d: stopping tablet replica
I20260927 07:08:24.395411 7649 raft_consensus.cc:2229] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:24.395462 7650 raft_consensus.cc:2229] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:24.395586 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7213
I20260927 07:08:24.395640 7650 raft_consensus.cc:2258] T 24c7cdad29e24e408fb150dc07c65432 P dc4fdcddff7942228a37f6b926c7650d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:24.397327 7648 ts_tablet_manager.cc:1905] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:24.399037 7648 ts_tablet_manager.cc:1918] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:24.399134 7648 log.cc:1196] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/24c7cdad29e24e408fb150dc07c65432
I20260927 07:08:24.395514 7649 raft_consensus.cc:2258] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:24.399484 7648 ts_tablet_manager.cc:1939] T 24c7cdad29e24e408fb150dc07c65432 P 90da36a179ce42e38a3a1318ae08534c: Deleting consensus metadata
I20260927 07:08:24.399917 7138 catalog_manager.cc:4841] TS 90da36a179ce42e38a3a1318ae08534c (127.21.2.66:36021): tablet 24c7cdad29e24e408fb150dc07c65432 (table test-table [id=56b822dcfe3f4ee4a22128a0adcb90ba]) successfully deleted
W20260927 07:08:24.400506 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:40015 (error 108)
I20260927 07:08:24.400743 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7347
I20260927 07:08:24.401413 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.400900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:40015 (local address 127.0.0.1:48340)
0927 07:08:24.401334 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.401373 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
W20260927 07:08:24.401660 7135 catalog_manager.cc:4584] TS dc4fdcddff7942228a37f6b926c7650d (127.21.2.65:40015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24c7cdad29e24e408fb150dc07c65432: Network error: Client connection negotiation failed: client connection to 127.21.2.65:40015: connect: Connection refused (error 111)
I20260927 07:08:24.403641 7649 ts_tablet_manager.cc:1905] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:24.404987 7649 ts_tablet_manager.cc:1918] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:24.405054 7649 log.cc:1196] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/24c7cdad29e24e408fb150dc07c65432
I20260927 07:08:24.405360 7649 ts_tablet_manager.cc:1939] T 24c7cdad29e24e408fb150dc07c65432 P 306242ed45b54edeb91e77a28eb710c3: Deleting consensus metadata
I20260927 07:08:24.405784 7137 catalog_manager.cc:4841] TS 306242ed45b54edeb91e77a28eb710c3 (127.21.2.67:35409): tablet 24c7cdad29e24e408fb150dc07c65432 (table test-table [id=56b822dcfe3f4ee4a22128a0adcb90ba]) successfully deleted
I20260927 07:08:24.406464 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7481
I20260927 07:08:24.411425 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7119
2026-09-27T07:08:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2080 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T07:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:24Z Disabled control of system clock
I20260927 07:08:24.439183 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42397
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:24.520347 7655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:24.520562 7655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:24.520583 7655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:24.521669 7655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:24.521703 7655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:24.521716 7655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:24.521735 7655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:24.521747 7655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:24.523100 7655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:42397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:24.523300 7655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:24.523608 7655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:24.528172 7660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.528172 7663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.528491 7655 server_base.cc:1034] running on GCE node
W20260927 07:08:24.528209 7661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.529029 7655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:24.529317 7655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:24.530464 7655 hybrid_clock.cc:648] HybridClock initialized: now 1790492904530449 us; error 27 us; skew 500 ppm
I20260927 07:08:24.531814 7655 webserver.cc:460] Webserver started at http://127.21.2.126:45997/ using document root <none> and password file <none>
I20260927 07:08:24.532084 7655 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:24.532132 7655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:24.532308 7655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:24.533408 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "b97c71741ce44484b2d14af1a8438783"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "7edf52cc2102bb4c3950a314cb5bfdcb"
server_key_iv: "d1f998703d7eb15170179e3047ae9daa"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.533776 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "b97c71741ce44484b2d14af1a8438783"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "7edf52cc2102bb4c3950a314cb5bfdcb"
server_key_iv: "d1f998703d7eb15170179e3047ae9daa"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.535086 7655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:24.535866 7669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.536046 7655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.536149 7655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "b97c71741ce44484b2d14af1a8438783"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "7edf52cc2102bb4c3950a314cb5bfdcb"
server_key_iv: "d1f998703d7eb15170179e3047ae9daa"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.536245 7655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:24.543814 7655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:24.544103 7655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:24.544260 7655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:24.548532 7655 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:42397
I20260927 07:08:24.548563 7721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:42397 every 8 connection(s)
I20260927 07:08:24.548991 7655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:24.550026 7722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:24.552523 7722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: Bootstrap starting.
I20260927 07:08:24.553149 7722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:24.553434 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7655
I20260927 07:08:24.553460 7722 log.cc:824] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:24.553653 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:24.553902 21513 external_mini_cluster.cc:1428] Setting key 54f578e60b289166137a893ee171d7e1
I20260927 07:08:24.554383 7722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: No bootstrap required, opened a new log
I20260927 07:08:24.556221 7722 raft_consensus.cc:348] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } }
I20260927 07:08:24.556388 7722 raft_consensus.cc:374] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:24.556432 7722 raft_consensus.cc:729] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97c71741ce44484b2d14af1a8438783, State: Initialized, Role: FOLLOWER
I20260927 07:08:24.556565 7722 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } }
I20260927 07:08:24.556655 7722 raft_consensus.cc:388] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:24.556701 7722 raft_consensus.cc:482] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:24.556773 7722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:24.557350 7722 raft_consensus.cc:504] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } }
I20260927 07:08:24.557472 7722 leader_election.cc:302] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b97c71741ce44484b2d14af1a8438783; no voters:
I20260927 07:08:24.557660 7722 leader_election.cc:288] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:24.557715 7727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:24.557852 7727 raft_consensus.cc:686] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [term 1 LEADER]: Becoming Leader. State: Replica: b97c71741ce44484b2d14af1a8438783, State: Running, Role: LEADER
I20260927 07:08:24.557928 7722 sys_catalog.cc:564] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:24.557936 7727 consensus_queue.cc:237] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } }
I20260927 07:08:24.558472 7728 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b97c71741ce44484b2d14af1a8438783. Latest consensus state: current_term: 1 leader_uuid: "b97c71741ce44484b2d14af1a8438783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } } }
I20260927 07:08:24.558559 7728 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:24.558712 7729 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b97c71741ce44484b2d14af1a8438783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97c71741ce44484b2d14af1a8438783" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 42397 } } }
I20260927 07:08:24.558777 7729 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:24.558761 7736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:24.559444 7736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:24.561558 7736 catalog_manager.cc:1347] Generated new cluster ID: a4c213435f67460eb6d52c9d47e15472
I20260927 07:08:24.561614 7736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:24.562067 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.554447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.21.2.126:42397)
0927 07:08:24.554708 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.554715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.554744 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:08:24.556203 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.556210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.556272 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.556418 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:24.556660 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.558370 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.559666 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.560064 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.560209 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.561313 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:24.561317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:24.561317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:24.561528 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:24.561572 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:24.561586 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:24.561679 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.561778 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.561862 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260927 07:08:24.593137 7736 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:24.593858 7736 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:24.606963 7736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: Generated new TSK 0
I20260927 07:08:24.607234 7736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:24.615811 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--builtin_ntp_servers=127.21.2.84:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:24.705245 7746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:24.705458 7746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:24.705482 7746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:24.706676 7746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:24.706718 7746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:24.706746 7746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:24.708178 7746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:24.708419 7746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:24.708740 7746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:24.713773 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.713783 7751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.713894 7746 server_base.cc:1034] running on GCE node
W20260927 07:08:24.713953 7754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.714208 7746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:24.714455 7746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:24.715595 7746 hybrid_clock.cc:648] HybridClock initialized: now 1790492904715581 us; error 30 us; skew 500 ppm
I20260927 07:08:24.716934 7746 webserver.cc:460] Webserver started at http://127.21.2.65:35719/ using document root <none> and password file <none>
I20260927 07:08:24.717211 7746 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:24.717260 7746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:24.717433 7746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:24.718523 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "b275cae9789841b88229fc29243177de"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "6477f33acbb5e8ea8cf47631da419611"
server_key_iv: "880978f05d6e476e6eb704b590198d1e"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.718885 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "b275cae9789841b88229fc29243177de"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "6477f33acbb5e8ea8cf47631da419611"
server_key_iv: "880978f05d6e476e6eb704b590198d1e"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.720240 7746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:24.720950 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.721117 7746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:24.721225 7746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "b275cae9789841b88229fc29243177de"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "6477f33acbb5e8ea8cf47631da419611"
server_key_iv: "880978f05d6e476e6eb704b590198d1e"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.721307 7746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:24.742503 7746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:24.742837 7746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:24.742990 7746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:24.743230 7746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:24.743646 7746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:24.743703 7746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.743748 7746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:24.743777 7746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.750525 7746 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:45467
I20260927 07:08:24.750571 7873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:45467 every 8 connection(s)
I20260927 07:08:24.750984 7746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:24.751812 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7746
I20260927 07:08:24.751917 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:24.752177 21513 external_mini_cluster.cc:1428] Setting key 4e5dd910e19fc2c0a6de5c1bf06bbc3b
I20260927 07:08:24.753274 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--builtin_ntp_servers=127.21.2.84:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:08:24.757015 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.751932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:59647 (local address 127.21.2.126:42397)
0927 07:08:24.751993 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.751995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.752287 (+ 292us) server_negotiation.cc:407] Connection header received
0927 07:08:24.752814 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.752815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.752827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.752896 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:24.753871 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.754363 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.755458 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.755673 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.755771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.756691 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:24.756692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:24.756692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:24.756701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:24.756718 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:24.756722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:24.756797 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.756899 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.756933 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:08:24.757266 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.751895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42397 (local address 127.21.2.65:59647)
0927 07:08:24.752124 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.752174 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.752737 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.752992 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.752996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.753007 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:24.753764 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.753772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.754482 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.754483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.755302 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.755305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.755781 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.755783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.755919 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.756517 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:24.756517 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:24.756548 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:24.756572 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:24.756797 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:24.756804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:24.756807 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.756888 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.757000 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260927 07:08:24.757866 7874 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42397
I20260927 07:08:24.757987 7874 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:24.758188 7874 heartbeater.cc:511] Master 127.21.2.126:42397 requested a full tablet report, sending...
I20260927 07:08:24.758759 7686 ts_manager.cc:194] Registered new tserver with Master: b275cae9789841b88229fc29243177de (127.21.2.65:45467)
I20260927 07:08:24.759490 7686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.65:59647
W20260927 07:08:24.836036 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:24.836246 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:24.836267 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:24.837443 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:24.837486 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:24.837512 7877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:24.838994 7877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:24.839241 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:24.839567 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:24.844645 7877 server_base.cc:1034] running on GCE node
W20260927 07:08:24.844623 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.844623 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.844623 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.845608 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:24.845894 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:24.847150 7877 hybrid_clock.cc:648] HybridClock initialized: now 1790492904847089 us; error 111 us; skew 500 ppm
I20260927 07:08:24.848827 7877 webserver.cc:460] Webserver started at http://127.21.2.66:37641/ using document root <none> and password file <none>
I20260927 07:08:24.849102 7877 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:24.849176 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:24.849336 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:24.850479 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "5fc4224572dd458290861744380fee47"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "a7aa183415b0b57392c9e7f14ae679f8"
server_key_iv: "0a3076885eb9783d552b7c4034bb0373"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.850872 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "5fc4224572dd458290861744380fee47"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "a7aa183415b0b57392c9e7f14ae679f8"
server_key_iv: "0a3076885eb9783d552b7c4034bb0373"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.852332 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 07:08:24.853133 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.853312 7877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:24.853420 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "5fc4224572dd458290861744380fee47"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "a7aa183415b0b57392c9e7f14ae679f8"
server_key_iv: "0a3076885eb9783d552b7c4034bb0373"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.853498 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:24.864359 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:24.864681 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:24.864845 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:24.865093 7877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:24.865428 7877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:24.865483 7877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.865523 7877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:24.865550 7877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.871326 7877 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:44981
I20260927 07:08:24.871425 8004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:44981 every 8 connection(s)
I20260927 07:08:24.871768 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:24.876886 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.872576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:37919 (local address 127.21.2.126:42397)
0927 07:08:24.872647 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:24.872649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:24.872860 (+ 211us) server_negotiation.cc:407] Connection header received
0927 07:08:24.873355 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:24.873356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:24.873367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:24.873434 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:24.874079 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.874573 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.875599 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.875791 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.875847 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.876635 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:24.876636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:24.876637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:24.876647 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:24.876661 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:24.876668 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:24.876718 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:24.876802 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 07:08:24.876826 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:24.877058 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:24.872522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42397 (local address 127.21.2.66:37919)
0927 07:08:24.872718 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:24.872761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:24.873253 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:24.873507 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:24.873510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:24.873518 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:24.874001 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.874005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.874659 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.874660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.875524 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:24.875527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:24.875865 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:24.875866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:24.875974 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:24.876494 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:24.876494 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:24.876521 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:24.876542 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:24.876725 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:24.876729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:24.876732 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:24.876783 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:08:24.876860 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260927 07:08:24.877573 8005 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42397
I20260927 07:08:24.877656 8005 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:24.877866 8005 heartbeater.cc:511] Master 127.21.2.126:42397 requested a full tablet report, sending...
I20260927 07:08:24.878266 7686 ts_manager.cc:194] Registered new tserver with Master: 5fc4224572dd458290861744380fee47 (127.21.2.66:44981)
I20260927 07:08:24.878454 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 7877
I20260927 07:08:24.878566 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:24.878718 7686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.66:37919
I20260927 07:08:24.878746 21513 external_mini_cluster.cc:1428] Setting key 8d80321e3f9a9f59b8e3cddb60cc53d2
I20260927 07:08:24.879666 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--builtin_ntp_servers=127.21.2.84:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:24.961549 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:24.961757 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:24.961779 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:24.962901 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:24.962935 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:24.962960 8008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:24.964395 8008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:24.964578 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:24.964840 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:24.969687 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.969698 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:24.969698 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:24.969931 8008 server_base.cc:1034] running on GCE node
I20260927 07:08:24.970249 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:24.970471 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:24.971606 8008 hybrid_clock.cc:648] HybridClock initialized: now 1790492904971595 us; error 26 us; skew 500 ppm
I20260927 07:08:24.972898 8008 webserver.cc:460] Webserver started at http://127.21.2.67:36399/ using document root <none> and password file <none>
I20260927 07:08:24.973135 8008 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:24.973215 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:24.973368 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:24.974488 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "7b60d46818be47f2a79f5549345602ff"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "2904988f40ca21102b999b233ab19abd"
server_key_iv: "1b9a088666eeb062ff7b22df91e089b6"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.974851 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "7b60d46818be47f2a79f5549345602ff"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "2904988f40ca21102b999b233ab19abd"
server_key_iv: "1b9a088666eeb062ff7b22df91e089b6"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.976210 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:24.976867 8022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.977046 8008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:24.977144 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "7b60d46818be47f2a79f5549345602ff"
format_stamp: "Formatted at 2026-09-27 07:08:24 on dist-test-slave-ltqz"
server_key: "2904988f40ca21102b999b233ab19abd"
server_key_iv: "1b9a088666eeb062ff7b22df91e089b6"
server_key_version: "encryptionkey@0"
I20260927 07:08:24.977232 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:24.994916 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:24.995265 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:24.995465 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:24.995705 8008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:24.996049 8008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:24.996106 8008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:24.996150 8008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:24.996183 8008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:25.002086 8008 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:37381
I20260927 07:08:25.002137 8135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:37381 every 8 connection(s)
I20260927 07:08:25.002516 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:25.004557 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8008
I20260927 07:08:25.004642 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:25.004850 21513 external_mini_cluster.cc:1428] Setting key 032eb2a56ae00b3a01b3b109109bb097
I20260927 07:08:25.007815 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.003247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:43793 (local address 127.21.2.126:42397)
0927 07:08:25.003322 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.003324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.003615 (+ 291us) server_negotiation.cc:407] Connection header received
0927 07:08:25.004147 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.004148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.004161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.004213 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:25.004980 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.005483 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.006455 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.006671 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.006764 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.007524 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:25.007526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:25.007526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:25.007530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:25.007544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:25.007549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:25.007603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.007698 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.007736 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 07:08:25.008013 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.003242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:42397 (local address 127.21.2.67:43793)
0927 07:08:25.003492 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.003525 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.004027 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.004281 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.004285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.004295 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:25.004872 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.004897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.005580 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.005581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.006361 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.006363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.006774 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.006775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.006910 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.007387 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:25.007387 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:25.007414 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:25.007434 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:25.007606 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:25.007610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:25.007613 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.007699 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.007794 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":171,"threads_started":1}
I20260927 07:08:25.008490 8136 heartbeater.cc:347] Connected to a master server at 127.21.2.126:42397
I20260927 07:08:25.008564 8136 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:25.008761 8136 heartbeater.cc:511] Master 127.21.2.126:42397 requested a full tablet report, sending...
I20260927 07:08:25.009125 7686 ts_manager.cc:194] Registered new tserver with Master: 7b60d46818be47f2a79f5549345602ff (127.21.2.67:37381)
I20260927 07:08:25.009464 7686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.67:43793
I20260927 07:08:25.016525 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:25.021098 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.017951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.21.2.126:42397)
0927 07:08:25.018036 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.018038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.018126 (+ 88us) server_negotiation.cc:407] Connection header received
0927 07:08:25.018196 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.018197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.018208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.018289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:25.018473 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.018944 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.019782 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.019968 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.020055 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.020771 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:25.020773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:25.020773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:25.020777 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:25.020790 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:25.020794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:25.020911 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.021001 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.021013 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:08:25.022869 7686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:25.024591 7686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:25.030768 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.026828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.21.2.65:45467)
0927 07:08:25.027162 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.027167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.027189 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:08:25.027222 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.027223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.027235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.027321 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:25.027504 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.028299 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.029646 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.030363 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.030434 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.030516 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.030652 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.030691 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260927 07:08:25.030761 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.026899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:44981 (local address 127.0.0.1:45962)
0927 07:08:25.027014 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.027025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.027078 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.027459 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.027461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.027463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:25.027507 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.027509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.028832 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.028833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.029848 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.029850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.030486 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.030487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.030541 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.030559 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.030605 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.030661 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":5,"thread_start_us":62,"threads_started":1}
I20260927 07:08:25.030761 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.026739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:45467 (local address 127.0.0.1:48224)
0927 07:08:25.026993 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.027002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.027043 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.027326 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.027329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.027331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:25.027418 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.027420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.028395 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.028396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.029550 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.029553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.030453 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.030454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.030541 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.030559 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.030630 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.030661 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260927 07:08:25.031183 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.027096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.21.2.66:44981)
0927 07:08:25.027254 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.027259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.027269 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:25.027311 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.027313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.027330 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.027584 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:25.027640 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.028741 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.029926 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.030376 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.030959 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.031061 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.031092 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.031124 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260927 07:08:25.031826 7808 tablet_service.cc:1467] Processing CreateTablet for tablet 63318e4303a64471850b8c8ff2a083a1 (DEFAULT_TABLE table=test-table [id=e2ee4f2b5af743feb810f7d1242bf5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:25.031837 7939 tablet_service.cc:1467] Processing CreateTablet for tablet 63318e4303a64471850b8c8ff2a083a1 (DEFAULT_TABLE table=test-table [id=e2ee4f2b5af743feb810f7d1242bf5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:25.032058 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.026712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:37381 (local address 127.0.0.1:54440)
0927 07:08:25.026911 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.026952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.027024 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.027488 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.027488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.027490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:25.027524 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.027525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.029934 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.029935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.030919 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.030922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.031849 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.031850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.031915 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.031919 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.031950 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.031990 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":53,"threads_started":1}
I20260927 07:08:25.032269 7939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63318e4303a64471850b8c8ff2a083a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:25.032269 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63318e4303a64471850b8c8ff2a083a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:25.032502 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.026799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.21.2.67:37381)
0927 07:08:25.027310 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.027313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.027322 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:25.027359 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.027379 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.027394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.027592 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:25.027643 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.028741 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.031012 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.031747 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.032271 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.032360 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.032389 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.032427 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":80,"threads_started":1}
I20260927 07:08:25.033107 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 63318e4303a64471850b8c8ff2a083a1 (DEFAULT_TABLE table=test-table [id=e2ee4f2b5af743feb810f7d1242bf5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:25.033427 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63318e4303a64471850b8c8ff2a083a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:25.035054 8150 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Bootstrap starting.
I20260927 07:08:25.035641 8151 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: Bootstrap starting.
I20260927 07:08:25.035867 8150 tablet_bootstrap.cc:654] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:25.036192 8150 log.cc:824] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:25.036415 8151 tablet_bootstrap.cc:654] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:25.036734 8151 log.cc:824] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:25.036734 8153 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Bootstrap starting.
I20260927 07:08:25.036935 8150 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: No bootstrap required, opened a new log
I20260927 07:08:25.037017 8150 ts_tablet_manager.cc:1397] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:08:25.037343 8151 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: No bootstrap required, opened a new log
I20260927 07:08:25.037338 8153 tablet_bootstrap.cc:654] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:25.037423 8151 ts_tablet_manager.cc:1397] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:25.037634 8153 log.cc:824] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:25.038816 8153 tablet_bootstrap.cc:492] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: No bootstrap required, opened a new log
I20260927 07:08:25.038885 8153 ts_tablet_manager.cc:1397] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:25.039686 8150 raft_consensus.cc:348] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.039891 8150 raft_consensus.cc:374] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:25.039932 8150 raft_consensus.cc:729] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc4224572dd458290861744380fee47, State: Initialized, Role: FOLLOWER
I20260927 07:08:25.039963 8151 raft_consensus.cc:348] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.040067 8150 consensus_queue.cc:260] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.040114 8151 raft_consensus.cc:374] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:25.040161 8151 raft_consensus.cc:729] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b275cae9789841b88229fc29243177de, State: Initialized, Role: FOLLOWER
I20260927 07:08:25.040272 8151 consensus_queue.cc:260] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.040369 8150 ts_tablet_manager.cc:1428] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 07:08:25.040489 8005 heartbeater.cc:503] Master 127.21.2.126:42397 was elected leader, sending a full tablet report...
I20260927 07:08:25.040535 8151 ts_tablet_manager.cc:1428] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:25.040629 7874 heartbeater.cc:503] Master 127.21.2.126:42397 was elected leader, sending a full tablet report...
I20260927 07:08:25.041880 8153 raft_consensus.cc:348] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.041993 8153 raft_consensus.cc:374] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:25.042013 8153 raft_consensus.cc:729] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b60d46818be47f2a79f5549345602ff, State: Initialized, Role: FOLLOWER
I20260927 07:08:25.042081 8153 consensus_queue.cc:260] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.042322 8153 ts_tablet_manager.cc:1428] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:08:25.042382 8136 heartbeater.cc:503] Master 127.21.2.126:42397 was elected leader, sending a full tablet report...
I20260927 07:08:25.054368 8156 raft_consensus.cc:482] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:25.054533 8156 raft_consensus.cc:504] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.054876 8156 leader_election.cc:288] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b60d46818be47f2a79f5549345602ff (127.21.2.67:37381), b275cae9789841b88229fc29243177de (127.21.2.65:45467)
I20260927 07:08:25.058801 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.055197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:35515 (local address 127.21.2.65:45467)
0927 07:08:25.055246 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.055248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.055281 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:08:25.055348 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.055349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.055358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.055450 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:25.055577 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.056261 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.057774 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.058365 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.058438 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.058447 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.058635 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.058701 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:08:25.058796 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.055008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:37381 (local address 127.21.2.66:38769)
0927 07:08:25.055051 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.055064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.055130 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.055291 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.055293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.055294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:25.055401 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.055403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.056330 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.056331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.057666 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.057670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.058483 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.058483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.058551 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.058558 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.058617 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.058672 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":34}
I20260927 07:08:25.058802 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.055052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:45467 (local address 127.21.2.66:35515)
0927 07:08:25.055227 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.055237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:25.055293 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:25.055451 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:25.055452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:25.055453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:25.055512 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.055513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.056330 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.056331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.057666 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:25.057670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.058466 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.058468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:25.058551 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.058558 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:08:25.058617 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:08:25.058669 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":72,"thread_start_us":95,"threads_started":1}
I20260927 07:08:25.059067 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.055090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:38769 (local address 127.21.2.67:37381)
0927 07:08:25.055160 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.055162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.055172 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:25.055202 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.055203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.055213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.055265 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:25.055459 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.056254 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.057775 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.058377 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.058437 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.058447 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.058920 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.058984 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:25.059224 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63318e4303a64471850b8c8ff2a083a1" candidate_uuid: "5fc4224572dd458290861744380fee47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b275cae9789841b88229fc29243177de" is_pre_election: true
I20260927 07:08:25.059468 7828 raft_consensus.cc:2454] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fc4224572dd458290861744380fee47 in term 0.
I20260927 07:08:25.059549 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63318e4303a64471850b8c8ff2a083a1" candidate_uuid: "5fc4224572dd458290861744380fee47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b60d46818be47f2a79f5549345602ff" is_pre_election: true
I20260927 07:08:25.059718 8090 raft_consensus.cc:2454] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fc4224572dd458290861744380fee47 in term 0.
I20260927 07:08:25.059707 7894 leader_election.cc:302] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fc4224572dd458290861744380fee47, b275cae9789841b88229fc29243177de; no voters:
I20260927 07:08:25.059834 8156 raft_consensus.cc:2781] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:25.059902 8156 raft_consensus.cc:482] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:25.059937 8156 raft_consensus.cc:3037] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:25.060626 8156 raft_consensus.cc:504] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.060783 8156 leader_election.cc:288] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [CANDIDATE]: Term 1 election: Requested vote from peers 7b60d46818be47f2a79f5549345602ff (127.21.2.67:37381), b275cae9789841b88229fc29243177de (127.21.2.65:45467)
I20260927 07:08:25.060920 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63318e4303a64471850b8c8ff2a083a1" candidate_uuid: "5fc4224572dd458290861744380fee47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b275cae9789841b88229fc29243177de"
I20260927 07:08:25.060943 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63318e4303a64471850b8c8ff2a083a1" candidate_uuid: "5fc4224572dd458290861744380fee47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b60d46818be47f2a79f5549345602ff"
I20260927 07:08:25.060993 7828 raft_consensus.cc:3037] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:25.061053 8090 raft_consensus.cc:3037] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:25.061708 7828 raft_consensus.cc:2454] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fc4224572dd458290861744380fee47 in term 1.
I20260927 07:08:25.061790 8090 raft_consensus.cc:2454] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fc4224572dd458290861744380fee47 in term 1.
I20260927 07:08:25.061862 7894 leader_election.cc:302] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fc4224572dd458290861744380fee47, b275cae9789841b88229fc29243177de; no voters:
I20260927 07:08:25.061945 8156 raft_consensus.cc:2781] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:25.062136 8156 raft_consensus.cc:686] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc4224572dd458290861744380fee47, State: Running, Role: LEADER
I20260927 07:08:25.062258 8156 consensus_queue.cc:237] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } }
I20260927 07:08:25.063026 7686 catalog_manager.cc:5495] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fc4224572dd458290861744380fee47 (127.21.2.66). New cstate: current_term: 1 leader_uuid: "5fc4224572dd458290861744380fee47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fc4224572dd458290861744380fee47" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 44981 } health_report { overall_health: HEALTHY } } }
I20260927 07:08:25.076664 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.074245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.21.2.66:44981)
0927 07:08:25.074315 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:25.074321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:25.074329 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:25.074359 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:25.074360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:25.074379 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:25.074457 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:25.074603 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.075068 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.075899 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:25.076092 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:25.076177 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:25.076319 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:25.076405 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:25.076517 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:25.076549 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 07:08:25.076558 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:25.078589 7828 raft_consensus.cc:1261] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 1 FOLLOWER]: Refusing update from remote peer 5fc4224572dd458290861744380fee47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:25.078640 8090 raft_consensus.cc:1261] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 1 FOLLOWER]: Refusing update from remote peer 5fc4224572dd458290861744380fee47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:25.078969 8156 consensus_queue.cc:1035] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b275cae9789841b88229fc29243177de" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:25.079138 8160 consensus_queue.cc:1035] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b60d46818be47f2a79f5549345602ff" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:25.081636 8163 mvcc.cc:205] Tried to move back new op lower bound from 7333858939198713856 to 7333858939135545344. Current Snapshot: MvccSnapshot[applied={T|T < 7333858939198713856}]
I20260927 07:08:25.082221 8165 mvcc.cc:205] Tried to move back new op lower bound from 7333858939198713856 to 7333858939135545344. Current Snapshot: MvccSnapshot[applied={T|T < 7333858939198713856}]
I20260927 07:08:25.082065 8164 mvcc.cc:205] Tried to move back new op lower bound from 7333858939198713856 to 7333858939135545344. Current Snapshot: MvccSnapshot[applied={T|T < 7333858939198713856}]
I20260927 07:08:25.084372 7685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:25.084477 7685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:25.085323 7685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b97c71741ce44484b2d14af1a8438783: Sending DeleteTablet for 3 replicas of tablet 63318e4303a64471850b8c8ff2a083a1
I20260927 07:08:25.085731 7808 tablet_service.cc:1514] Processing DeleteTablet for tablet 63318e4303a64471850b8c8ff2a083a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:25 UTC) from {username='slave'} at 127.0.0.1:48224
I20260927 07:08:25.085831 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 63318e4303a64471850b8c8ff2a083a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:25 UTC) from {username='slave'} at 127.0.0.1:54440
I20260927 07:08:25.085882 7939 tablet_service.cc:1514] Processing DeleteTablet for tablet 63318e4303a64471850b8c8ff2a083a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:25 UTC) from {username='slave'} at 127.0.0.1:45962
I20260927 07:08:25.085987 8173 tablet_replica.cc:330] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de: stopping tablet replica
I20260927 07:08:25.086090 8173 raft_consensus.cc:2229] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:25.086205 8173 raft_consensus.cc:2258] T 63318e4303a64471850b8c8ff2a083a1 P b275cae9789841b88229fc29243177de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:25.086337 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7746
I20260927 07:08:25.088286 8174 tablet_replica.cc:330] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: stopping tablet replica
I20260927 07:08:25.088407 8174 raft_consensus.cc:2229] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:25.088515 8174 raft_consensus.cc:2258] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:25.088799 8175 tablet_replica.cc:330] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: stopping tablet replica
I20260927 07:08:25.088907 8175 raft_consensus.cc:2229] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:25.089032 8175 raft_consensus.cc:2258] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:25.089337 8174 ts_tablet_manager.cc:1905] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:25.089457 8175 ts_tablet_manager.cc:1905] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:25.091130 8174 ts_tablet_manager.cc:1918] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:25.091290 8174 log.cc:1196] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/63318e4303a64471850b8c8ff2a083a1
I20260927 07:08:25.091451 8175 ts_tablet_manager.cc:1918] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:25.091534 8175 log.cc:1196] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/63318e4303a64471850b8c8ff2a083a1
I20260927 07:08:25.091792 8175 ts_tablet_manager.cc:1939] T 63318e4303a64471850b8c8ff2a083a1 P 5fc4224572dd458290861744380fee47: Deleting consensus metadata
I20260927 07:08:25.092151 8174 ts_tablet_manager.cc:1939] T 63318e4303a64471850b8c8ff2a083a1 P 7b60d46818be47f2a79f5549345602ff: Deleting consensus metadata
I20260927 07:08:25.092444 7673 catalog_manager.cc:4841] TS 5fc4224572dd458290861744380fee47 (127.21.2.66:44981): tablet 63318e4303a64471850b8c8ff2a083a1 (table test-table [id=e2ee4f2b5af743feb810f7d1242bf5c3]) successfully deleted
I20260927 07:08:25.092684 7673 catalog_manager.cc:4841] TS 7b60d46818be47f2a79f5549345602ff (127.21.2.67:37381): tablet 63318e4303a64471850b8c8ff2a083a1 (table test-table [id=e2ee4f2b5af743feb810f7d1242bf5c3]) successfully deleted
W20260927 07:08:25.093031 7672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:45467 (error 108)
I20260927 07:08:25.093245 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7877
I20260927 07:08:25.093883 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:25.093357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:45467 (local address 127.0.0.1:48236)
0927 07:08:25.093788 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:25.093841 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
W20260927 07:08:25.095582 7672 catalog_manager.cc:4584] TS b275cae9789841b88229fc29243177de (127.21.2.65:45467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63318e4303a64471850b8c8ff2a083a1: Network error: Client connection negotiation failed: client connection to 127.21.2.65:45467: connect: Connection refused (error 111)
I20260927 07:08:25.098882 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8008
I20260927 07:08:25.103451 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 7655
2026-09-27T07:08:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (690 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:25 dist-test-slave-ltqz krb5kdc[8179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:25 dist-test-slave-ltqz krb5kdc[8179](info): set up 2 sockets
Sep 27 07:08:25 dist-test-slave-ltqz krb5kdc[8179](info): commencing operation
krb5kdc: starting...
W20260927 07:08:26.135305 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:26.253450 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43661
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:26.334048 8192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:26.334257 8192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:26.334280 8192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:26.335342 8192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:26.335394 8192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:26.335408 8192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:26.335421 8192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:26.335433 8192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:26.336786 8192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:43661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:26.336966 8192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:26.337126 8192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:26.341877 8192 server_base.cc:1034] running on GCE node
W20260927 07:08:26.341739 8200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.341797 8198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.341739 8197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:26.342244 8192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:26.342543 8192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:26.343703 8192 hybrid_clock.cc:648] HybridClock initialized: now 1790492906343690 us; error 47 us; skew 500 ppm
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:26.346251 8192 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:26.346755 8192 webserver.cc:460] Webserver started at http://127.21.2.126:41369/ using document root <none> and password file <none>
I20260927 07:08:26.346989 8192 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:26.347062 8192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:26.347215 8192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:26.348357 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "8384c14c2942467286284a94a13df616"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "b2423c6c114d92e296b87e81fc39d3af"
server_key_iv: "cb5da7fc912cea7fe93fa07c62b71778"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.348740 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "8384c14c2942467286284a94a13df616"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "b2423c6c114d92e296b87e81fc39d3af"
server_key_iv: "cb5da7fc912cea7fe93fa07c62b71778"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.350096 8192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:26.350858 8207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.351046 8192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:26.351147 8192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "8384c14c2942467286284a94a13df616"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "b2423c6c114d92e296b87e81fc39d3af"
server_key_iv: "cb5da7fc912cea7fe93fa07c62b71778"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.351238 8192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:26.379549 8192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:26.381493 8192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:26.381665 8192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:26.386138 8192 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:43661
I20260927 07:08:26.386183 8259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:43661 every 8 connection(s)
I20260927 07:08:26.386547 8192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:26.387470 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:26.388403 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8192
I20260927 07:08:26.388545 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:26.388736 21513 external_mini_cluster.cc:1428] Setting key 986816463b67b8c8bc9254abd613f985
I20260927 07:08:26.390473 8260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: Bootstrap starting.
I20260927 07:08:26.391150 8260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:26.391510 8260 log.cc:824] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:26.392262 8260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: No bootstrap required, opened a new log
I20260927 07:08:26.394599 8260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } }
I20260927 07:08:26.394744 8260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:26.394767 8260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8384c14c2942467286284a94a13df616, State: Initialized, Role: FOLLOWER
I20260927 07:08:26.394905 8260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } }
I20260927 07:08:26.395001 8260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:26.395025 8260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:26.395095 8260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:26.395726 8260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } }
I20260927 07:08:26.395849 8260 leader_election.cc:302] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8384c14c2942467286284a94a13df616; no voters:
I20260927 07:08:26.396028 8260 leader_election.cc:288] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:26.396061 8265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:26.396185 8265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [term 1 LEADER]: Becoming Leader. State: Replica: 8384c14c2942467286284a94a13df616, State: Running, Role: LEADER
I20260927 07:08:26.396260 8265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } }
I20260927 07:08:26.396297 8260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:26.396591 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8384c14c2942467286284a94a13df616. Latest consensus state: current_term: 1 leader_uuid: "8384c14c2942467286284a94a13df616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } } }
I20260927 07:08:26.396667 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:26.396612 8266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8384c14c2942467286284a94a13df616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8384c14c2942467286284a94a13df616" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 43661 } } }
I20260927 07:08:26.396883 8266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616 [sys.catalog]: This master's current role is: LEADER
W20260927 07:08:26.398058 8280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:26.398145 8280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:26.398221 8269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:26.398401 8269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:26.399842 8269 catalog_manager.cc:1347] Generated new cluster ID: 2ca942efd6d446c99af779f475cd0137
I20260927 07:08:26.399897 8269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:26.400727 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.389311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.21.2.126:43661)
0927 07:08:26.389527 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.389535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.389568 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:08:26.390189 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.390198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.390260 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.390413 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:26.391068 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.392492 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.393236 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.393865 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.394000 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.396748 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:26.396753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:26.396753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:26.396766 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:26.398782 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.399204 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.399207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.399207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.399305 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.399537 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.399539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.399539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.399843 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:26.399966 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.400386 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.400521 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260927 07:08:26.412920 8269 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:26.413667 8269 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:26.422478 8269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: Generated new TSK 0
I20260927 07:08:26.422698 8269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
Entry for principal kudu/127.21.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab.
I20260927 07:08:26.454254 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:26.535519 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:26.535734 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:26.535758 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:26.536854 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:26.536890 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:26.536917 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:26.538435 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.65.keytab
--principal=kudu/127.21.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:26.538628 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:26.538838 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:26.543637 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.543748 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.543648 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:26.544059 8286 server_base.cc:1034] running on GCE node
I20260927 07:08:26.544270 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:26.544510 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:26.545699 8286 hybrid_clock.cc:648] HybridClock initialized: now 1790492906545689 us; error 33 us; skew 500 ppm
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:26.548506 8286 init.cc:377] Logged in from keytab as kudu/127.21.2.65@KRBTEST.COM (short username kudu)
I20260927 07:08:26.548954 8286 webserver.cc:460] Webserver started at http://127.21.2.65:44643/ using document root <none> and password file <none>
I20260927 07:08:26.549196 8286 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:26.549266 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:26.549417 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:26.550493 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "0eff13f250e64fb993340607c8a98b25"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "ca644a43ecdf0dd9ed5102baac790580"
server_key_iv: "37b24814497fc63a4faeba2eb88437d4"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.550869 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "0eff13f250e64fb993340607c8a98b25"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "ca644a43ecdf0dd9ed5102baac790580"
server_key_iv: "37b24814497fc63a4faeba2eb88437d4"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.552284 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:26.553051 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.553310 8286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:26.553418 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "0eff13f250e64fb993340607c8a98b25"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "ca644a43ecdf0dd9ed5102baac790580"
server_key_iv: "37b24814497fc63a4faeba2eb88437d4"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.553515 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:26.572669 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:26.575160 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:26.575330 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:26.575670 8286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:26.575994 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:26.576056 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.576100 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:26.576131 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.581943 8286 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:41025
I20260927 07:08:26.582042 8414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:41025 every 8 connection(s)
I20260927 07:08:26.582366 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.65@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:26.589764 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8286
I20260927 07:08:26.589895 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:26.590130 21513 external_mini_cluster.cc:1428] Setting key e04e6069c6f527f3c77b289086532faa
I20260927 07:08:26.592352 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.583178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:54835 (local address 127.21.2.126:43661)
0927 07:08:26.583259 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.583261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.583486 (+ 225us) server_negotiation.cc:407] Connection header received
0927 07:08:26.584027 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.584028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.584040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.584093 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:26.585084 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.585589 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.586709 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.586922 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.587001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.589204 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:26.589206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:26.589206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:26.589214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:26.590717 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.591233 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.591235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.591235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.591299 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.591639 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.591640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.591640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.591798 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:26.591874 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.592123 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.592219 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:08:26.592530 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.583142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43661 (local address 127.21.2.65:54835)
0927 07:08:26.583322 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.583389 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.583898 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.584172 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.584175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.584774 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:26.584977 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.584982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.585689 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.585690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.586614 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.586616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.587010 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.587011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.587141 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.587699 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:26.587707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:26.589100 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:26.590869 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.590878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.590880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.591135 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.591458 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.591459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.591460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.591565 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.591868 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:26.591872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:26.591986 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.592063 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.592182 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260927 07:08:26.593119 8415 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43661
I20260927 07:08:26.593225 8415 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:26.593421 8415 heartbeater.cc:511] Master 127.21.2.126:43661 requested a full tablet report, sending...
I20260927 07:08:26.593964 8224 ts_manager.cc:194] Registered new tserver with Master: 0eff13f250e64fb993340607c8a98b25 (127.21.2.65:41025)
I20260927 07:08:26.594569 8224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.65@KRBTEST.COM'} at 127.21.2.65:54835
WARNING: no policy specified for kudu/127.21.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
Entry for principal kudu/127.21.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab.
I20260927 07:08:26.614631 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:26.696909 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:26.697220 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:26.697269 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:26.699085 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:26.699165 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:26.699219 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:26.700892 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.66.keytab
--principal=kudu/127.21.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:26.701180 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:26.701421 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:26.706467 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.706465 8426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.706439 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:26.706729 8420 server_base.cc:1034] running on GCE node
I20260927 07:08:26.706960 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:26.707232 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:26.708374 8420 hybrid_clock.cc:648] HybridClock initialized: now 1790492906708367 us; error 31 us; skew 500 ppm
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:26.710835 8420 init.cc:377] Logged in from keytab as kudu/127.21.2.66@KRBTEST.COM (short username kudu)
I20260927 07:08:26.711347 8420 webserver.cc:460] Webserver started at http://127.21.2.66:44625/ using document root <none> and password file <none>
I20260927 07:08:26.711627 8420 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:26.711696 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:26.711833 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:26.712878 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "1da71ce482a7418e9028686305693d92"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "a3e52a2f52a991bd9d4b1f73f6deb283"
server_key_iv: "f44300eac4615486b327e797f34aa01f"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.713249 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "1da71ce482a7418e9028686305693d92"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "a3e52a2f52a991bd9d4b1f73f6deb283"
server_key_iv: "f44300eac4615486b327e797f34aa01f"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.714612 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:26.715349 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.715600 8420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:26.715698 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "1da71ce482a7418e9028686305693d92"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "a3e52a2f52a991bd9d4b1f73f6deb283"
server_key_iv: "f44300eac4615486b327e797f34aa01f"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.715786 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:26.734002 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:26.735903 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:26.736059 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:26.736300 8420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:26.736608 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:26.736665 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.736708 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:26.736733 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.742726 8420 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:37595
I20260927 07:08:26.742771 8548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:37595 every 8 connection(s)
I20260927 07:08:26.743072 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:26.749748 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8420
I20260927 07:08:26.749873 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:26.750195 21513 external_mini_cluster.cc:1428] Setting key 89cf00057883bb97b7613559dcf498a9
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.66@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:26.754613 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.743845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:51423 (local address 127.21.2.126:43661)
0927 07:08:26.743887 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.743889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.744111 (+ 222us) server_negotiation.cc:407] Connection header received
0927 07:08:26.745424 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.745426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.745438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.745495 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:26.746736 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.747392 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.748567 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.748799 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.748869 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.751345 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:26.751346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:26.751347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:26.751355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:26.752979 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.753519 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.753520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.753520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.753582 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.753875 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.753876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.753876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.754035 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:26.754100 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.754426 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.754519 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:26.754863 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.743818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43661 (local address 127.21.2.66:51423)
0927 07:08:26.743981 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.744024 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.744497 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.745561 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.745565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.746322 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:26.746544 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.746549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.747571 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.747572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.748437 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.748439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.748885 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.748886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.748991 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.749584 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:26.749595 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:26.751248 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:26.753069 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.753077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.753079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.753420 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.753653 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.753654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.753654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.753807 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.754106 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:26.754112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:26.754283 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.754401 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.754570 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260927 07:08:26.755332 8549 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43661
I20260927 07:08:26.755472 8549 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:26.755684 8549 heartbeater.cc:511] Master 127.21.2.126:43661 requested a full tablet report, sending...
I20260927 07:08:26.756109 8224 ts_manager.cc:194] Registered new tserver with Master: 1da71ce482a7418e9028686305693d92 (127.21.2.66:37595)
I20260927 07:08:26.756711 8224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.66@KRBTEST.COM'} at 127.21.2.66:51423
WARNING: no policy specified for kudu/127.21.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
Entry for principal kudu/127.21.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab.
I20260927 07:08:26.778023 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:26.859261 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:26.859591 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:26.859635 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:26.860816 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:26.860862 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:26.860888 8554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:26.862424 8554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.67.keytab
--principal=kudu/127.21.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:43661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:26.862679 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:26.862912 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:26.867795 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.867797 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:26.867795 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:26.868224 8554 server_base.cc:1034] running on GCE node
I20260927 07:08:26.868433 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:26.868680 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:26.869817 8554 hybrid_clock.cc:648] HybridClock initialized: now 1790492906869808 us; error 26 us; skew 500 ppm
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:26.872160 8554 init.cc:377] Logged in from keytab as kudu/127.21.2.67@KRBTEST.COM (short username kudu)
I20260927 07:08:26.872576 8554 webserver.cc:460] Webserver started at http://127.21.2.67:46199/ using document root <none> and password file <none>
I20260927 07:08:26.872803 8554 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:26.872871 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:26.873020 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:26.874027 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "b5f0992da98446d0ae437c293bafcc9d"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "0acf599254b6da93ef082308e49a1ec0"
server_key_iv: "6ed6be88ddddf854331e9ac58131a00a"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.874382 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "b5f0992da98446d0ae437c293bafcc9d"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "0acf599254b6da93ef082308e49a1ec0"
server_key_iv: "6ed6be88ddddf854331e9ac58131a00a"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.875717 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:26.876400 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.876569 8554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:26.876675 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "b5f0992da98446d0ae437c293bafcc9d"
format_stamp: "Formatted at 2026-09-27 07:08:26 on dist-test-slave-ltqz"
server_key: "0acf599254b6da93ef082308e49a1ec0"
server_key_iv: "6ed6be88ddddf854331e9ac58131a00a"
server_key_version: "encryptionkey@0"
I20260927 07:08:26.876747 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:26.889770 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:26.891680 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:26.891842 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:26.892119 8554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:26.892443 8554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:26.892498 8554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.892544 8554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:26.892570 8554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:26.899255 8554 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:45869
I20260927 07:08:26.899331 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:45869 every 8 connection(s)
I20260927 07:08:26.899663 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:26.903036 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8554
I20260927 07:08:26.903124 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:26.903308 21513 external_mini_cluster.cc:1428] Setting key 20e573b87e9cf0b9c5220922ceb034ea
Sep 27 07:08:26 dist-test-slave-ltqz krb5kdc[8179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492906, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.67@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:26.908713 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.900383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:50805 (local address 127.21.2.126:43661)
0927 07:08:26.900445 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.900447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.900639 (+ 192us) server_negotiation.cc:407] Connection header received
0927 07:08:26.901119 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.901120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.901131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.901185 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:26.902083 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.902582 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.903832 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.904099 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.904187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.906172 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:26.906173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:26.906173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:26.906180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:26.907617 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.907950 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.907951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.907951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.908001 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.908214 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.908214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.908214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.908312 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:26.908361 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.908585 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.908643 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:08:26.908933 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.900358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:43661 (local address 127.21.2.67:50805)
0927 07:08:26.900532 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.900567 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.901032 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.901247 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.901250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.901809 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:26.902004 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.902008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.902682 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.902683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.903719 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.903722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.904210 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.904211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.904354 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.904798 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:26.904806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:26.906084 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:26.907685 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.907690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.907692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.907887 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.908064 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:26.908064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:26.908065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:26.908159 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:26.908367 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:26.908371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:26.908481 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.908558 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.908682 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260927 07:08:26.909441 8683 heartbeater.cc:347] Connected to a master server at 127.21.2.126:43661
I20260927 07:08:26.909538 8683 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:26.909686 8683 heartbeater.cc:511] Master 127.21.2.126:43661 requested a full tablet report, sending...
I20260927 07:08:26.910053 8224 ts_manager.cc:194] Registered new tserver with Master: b5f0992da98446d0ae437c293bafcc9d (127.21.2.67:45869)
I20260927 07:08:26.910418 8224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.2.67@KRBTEST.COM'} at 127.21.2.67:50805
I20260927 07:08:26.915019 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:26.921730 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.916367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.21.2.126:43661)
0927 07:08:26.916429 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.916431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.916527 (+ 96us) server_negotiation.cc:407] Connection header received
0927 07:08:26.916565 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.916565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.916574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.916616 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:26.917113 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.917567 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.918246 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.918494 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.918540 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.919714 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:26.919716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:26.919716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:26.919722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:26.920822 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.921135 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.921136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.921136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.921186 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:26.921316 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:26.921317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:26.921317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:26.921446 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:26.921517 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.921617 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.921670 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 07:08:26.923586 8224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:26.925086 8224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:26.931572 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.21.2.65:41025)
0927 07:08:26.927744 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.927756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.927764 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:26.927795 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.927796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.927808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.927899 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:26.928221 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.928971 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.930379 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.931082 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.931178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.931243 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.931428 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.931483 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":83,"threads_started":1}
I20260927 07:08:26.931592 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:45869 (local address 127.0.0.1:42482)
0927 07:08:26.927493 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.927534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.927605 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.927919 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.927922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.927924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:26.928027 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.928029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.929399 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.929400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.930452 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.930454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.931303 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.931304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.931413 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.931419 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.931471 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.931512 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260927 07:08:26.931675 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.21.2.67:45869)
0927 07:08:26.927722 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.927727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.927739 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:08:26.927784 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.927786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.927806 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.928112 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:26.928205 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.929295 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.930525 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.931198 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.931295 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.931347 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.931551 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.931600 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260927 07:08:26.931499 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41025 (local address 127.0.0.1:34188)
0927 07:08:26.927570 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.927579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.927631 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.927990 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.927991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.927994 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:26.928134 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.928137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.929075 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.929076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.930299 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.930302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.931177 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.931178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.931259 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.931276 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.931344 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.931421 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4,"thread_start_us":138,"threads_started":1}
I20260927 07:08:26.932471 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37595 (local address 127.0.0.1:42130)
0927 07:08:26.927784 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:26.927794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:26.927862 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:26.928031 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:26.928032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:26.928033 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:26.928081 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.928083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.930520 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.930521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.931225 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:26.931226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.932251 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.932252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:26.932312 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.932316 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:26.932363 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:08:26.932406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":42,"threads_started":1}
I20260927 07:08:26.932395 8617 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 (DEFAULT_TABLE table=test-table [id=ef7132a741d14cd7b0464785c550caca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:26.932760 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:26.927543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.21.2.66:37595)
0927 07:08:26.927732 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:26.927736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:26.927880 (+ 144us) server_negotiation.cc:407] Connection header received
0927 07:08:26.927926 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:26.927928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:26.927942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:26.928017 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:26.928169 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.929289 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.931316 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:26.932087 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:26.932164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:26.932214 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:26.932623 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 07:08:26.932666 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260927 07:08:26.932778 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb59cb2249425b9eb46b6260e6e9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:26.932689 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 (DEFAULT_TABLE table=test-table [id=ef7132a741d14cd7b0464785c550caca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:26.933020 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb59cb2249425b9eb46b6260e6e9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:26.933326 8483 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 (DEFAULT_TABLE table=test-table [id=ef7132a741d14cd7b0464785c550caca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:26.933616 8483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb59cb2249425b9eb46b6260e6e9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:26.935966 8700 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Bootstrap starting.
I20260927 07:08:26.936173 8701 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Bootstrap starting.
I20260927 07:08:26.936694 8700 tablet_bootstrap.cc:654] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:26.936714 8702 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Bootstrap starting.
I20260927 07:08:26.936908 8701 tablet_bootstrap.cc:654] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:26.937289 8702 tablet_bootstrap.cc:654] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:26.937304 8700 log.cc:824] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:26.937469 8701 log.cc:824] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:26.937863 8702 log.cc:824] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:26.938108 8700 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: No bootstrap required, opened a new log
I20260927 07:08:26.938212 8700 ts_tablet_manager.cc:1397] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:08:26.938220 8701 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: No bootstrap required, opened a new log
I20260927 07:08:26.938283 8701 ts_tablet_manager.cc:1397] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:26.938591 8702 tablet_bootstrap.cc:492] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: No bootstrap required, opened a new log
I20260927 07:08:26.938669 8702 ts_tablet_manager.cc:1397] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:26.939829 8700 raft_consensus.cc:348] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.939988 8700 raft_consensus.cc:374] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:26.940034 8700 raft_consensus.cc:729] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eff13f250e64fb993340607c8a98b25, State: Initialized, Role: FOLLOWER
I20260927 07:08:26.940147 8700 consensus_queue.cc:260] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.940418 8700 ts_tablet_manager.cc:1428] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:08:26.940574 8415 heartbeater.cc:503] Master 127.21.2.126:43661 was elected leader, sending a full tablet report...
I20260927 07:08:26.940551 8702 raft_consensus.cc:348] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.940622 8701 raft_consensus.cc:348] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.940734 8702 raft_consensus.cc:374] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:26.940762 8701 raft_consensus.cc:374] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:26.940779 8702 raft_consensus.cc:729] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da71ce482a7418e9028686305693d92, State: Initialized, Role: FOLLOWER
I20260927 07:08:26.940795 8701 raft_consensus.cc:729] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f0992da98446d0ae437c293bafcc9d, State: Initialized, Role: FOLLOWER
I20260927 07:08:26.940898 8701 consensus_queue.cc:260] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.940898 8702 consensus_queue.cc:260] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:26.941159 8702 ts_tablet_manager.cc:1428] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:08:26.941170 8683 heartbeater.cc:503] Master 127.21.2.126:43661 was elected leader, sending a full tablet report...
I20260927 07:08:26.941265 8549 heartbeater.cc:503] Master 127.21.2.126:43661 was elected leader, sending a full tablet report...
I20260927 07:08:26.941391 8701 ts_tablet_manager.cc:1428] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260927 07:08:26.993686 8550 tablet.cc:2369] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:08:27.083153 8416 tablet.cc:2369] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:27.123899 8707 raft_consensus.cc:482] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:27.124037 8707 raft_consensus.cc:504] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:27.124393 8707 leader_election.cc:288] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eff13f250e64fb993340607c8a98b25 (127.21.2.65:41025), 1da71ce482a7418e9028686305693d92 (127.21.2.66:37595)
I20260927 07:08:27.128261 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.124614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:38263 (local address 127.21.2.65:41025)
0927 07:08:27.124669 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.124672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.125083 (+ 411us) server_negotiation.cc:407] Connection header received
0927 07:08:27.125175 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.125176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.125191 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.125250 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.125373 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.126034 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127302 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.127878 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127957 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.127970 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.128151 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.128190 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:08:27.128260 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.124719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:49041 (local address 127.21.2.66:37595)
0927 07:08:27.124809 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.124811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.124820 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:27.124851 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.124852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.124865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.124925 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.125139 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.125921 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127310 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.127908 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127968 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.127978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.128152 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.128190 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 07:08:27.128258 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.124585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37595 (local address 127.21.2.67:49041)
0927 07:08:27.124663 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.124675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.124734 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.124952 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.124954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.124956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.125051 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.125053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.126003 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.126003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.127210 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.127213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.127977 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.128059 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.128063 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.128135 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.128166 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":8}
I20260927 07:08:27.128268 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.124588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41025 (local address 127.21.2.67:38263)
0927 07:08:27.125006 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.125017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.125112 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.125262 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.125262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.125264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.125311 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.125313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.126114 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.126114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.127210 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.127213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.127977 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.127978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.128059 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.128063 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.128123 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.128166 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":216,"thread_start_us":124,"threads_started":1}
I20260927 07:08:27.128639 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb59cb2249425b9eb46b6260e6e9a8" candidate_uuid: "b5f0992da98446d0ae437c293bafcc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eff13f250e64fb993340607c8a98b25" is_pre_election: true
I20260927 07:08:27.128679 8503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb59cb2249425b9eb46b6260e6e9a8" candidate_uuid: "b5f0992da98446d0ae437c293bafcc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da71ce482a7418e9028686305693d92" is_pre_election: true
I20260927 07:08:27.128785 8369 raft_consensus.cc:2454] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5f0992da98446d0ae437c293bafcc9d in term 0.
I20260927 07:08:27.128856 8503 raft_consensus.cc:2454] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5f0992da98446d0ae437c293bafcc9d in term 0.
I20260927 07:08:27.128995 8573 leader_election.cc:302] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eff13f250e64fb993340607c8a98b25, b5f0992da98446d0ae437c293bafcc9d; no voters:
I20260927 07:08:27.129141 8707 raft_consensus.cc:2781] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:27.129201 8707 raft_consensus.cc:482] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:27.129231 8707 raft_consensus.cc:3037] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.129873 8707 raft_consensus.cc:504] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:27.130014 8707 leader_election.cc:288] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [CANDIDATE]: Term 1 election: Requested vote from peers 0eff13f250e64fb993340607c8a98b25 (127.21.2.65:41025), 1da71ce482a7418e9028686305693d92 (127.21.2.66:37595)
I20260927 07:08:27.130162 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb59cb2249425b9eb46b6260e6e9a8" candidate_uuid: "b5f0992da98446d0ae437c293bafcc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eff13f250e64fb993340607c8a98b25"
I20260927 07:08:27.130225 8503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb59cb2249425b9eb46b6260e6e9a8" candidate_uuid: "b5f0992da98446d0ae437c293bafcc9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da71ce482a7418e9028686305693d92"
I20260927 07:08:27.130244 8369 raft_consensus.cc:3037] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.130311 8503 raft_consensus.cc:3037] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.130980 8369 raft_consensus.cc:2454] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5f0992da98446d0ae437c293bafcc9d in term 1.
I20260927 07:08:27.131078 8503 raft_consensus.cc:2454] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5f0992da98446d0ae437c293bafcc9d in term 1.
I20260927 07:08:27.131148 8573 leader_election.cc:302] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eff13f250e64fb993340607c8a98b25, b5f0992da98446d0ae437c293bafcc9d; no voters:
I20260927 07:08:27.131250 8707 raft_consensus.cc:2781] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:27.131472 8707 raft_consensus.cc:686] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 1 LEADER]: Becoming Leader. State: Replica: b5f0992da98446d0ae437c293bafcc9d, State: Running, Role: LEADER
I20260927 07:08:27.131579 8707 consensus_queue.cc:237] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } }
I20260927 07:08:27.132402 8224 catalog_manager.cc:5495] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d reported cstate change: term changed from 0 to 1, leader changed from <none> to b5f0992da98446d0ae437c293bafcc9d (127.21.2.67). New cstate: current_term: 1 leader_uuid: "b5f0992da98446d0ae437c293bafcc9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5f0992da98446d0ae437c293bafcc9d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 } health_report { overall_health: UNKNOWN } } }
W20260927 07:08:27.150254 8684 tablet.cc:2369] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:27.161363 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.158871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.21.2.67:45869)
0927 07:08:27.158935 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.158937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.158946 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:27.158986 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.158987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.159015 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.159094 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:27.159255 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.159754 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.160541 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.160744 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.160821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.160986 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:27.161091 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:27.161186 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.161229 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.161250 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:08:27.163058 8369 raft_consensus.cc:1261] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 1 FOLLOWER]: Refusing update from remote peer b5f0992da98446d0ae437c293bafcc9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:27.163067 8503 raft_consensus.cc:1261] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 1 FOLLOWER]: Refusing update from remote peer b5f0992da98446d0ae437c293bafcc9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:27.163455 8710 consensus_queue.cc:1035] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eff13f250e64fb993340607c8a98b25" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:27.163601 8707 consensus_queue.cc:1035] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da71ce482a7418e9028686305693d92" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:27.166210 8713 mvcc.cc:205] Tried to move back new op lower bound from 7333858947737313280 to 7333858947611729920. Current Snapshot: MvccSnapshot[applied={T|T < 7333858947737313280}]
I20260927 07:08:27.166469 8715 mvcc.cc:205] Tried to move back new op lower bound from 7333858947737313280 to 7333858947611729920. Current Snapshot: MvccSnapshot[applied={T|T < 7333858947737313280}]
I20260927 07:08:27.168330 8224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:27.168430 8224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:27.169202 8224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8384c14c2942467286284a94a13df616: Sending DeleteTablet for 3 replicas of tablet 8beb59cb2249425b9eb46b6260e6e9a8
I20260927 07:08:27.169559 8483 tablet_service.cc:1514] Processing DeleteTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:42130
I20260927 07:08:27.169684 8617 tablet_service.cc:1514] Processing DeleteTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:42482
I20260927 07:08:27.169797 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 8beb59cb2249425b9eb46b6260e6e9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='kudu', principal='kudu/127.21.2.126@KRBTEST.COM'} at 127.0.0.1:34188
I20260927 07:08:27.169862 8723 tablet_replica.cc:330] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: stopping tablet replica
I20260927 07:08:27.170001 8725 tablet_replica.cc:330] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25: stopping tablet replica
I20260927 07:08:27.170094 8725 raft_consensus.cc:2229] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:27.170187 8723 raft_consensus.cc:2229] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:27.170186 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8286
I20260927 07:08:27.170220 8725 raft_consensus.cc:2258] T 8beb59cb2249425b9eb46b6260e6e9a8 P 0eff13f250e64fb993340607c8a98b25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:27.170356 8723 raft_consensus.cc:2258] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:27.171514 8723 ts_tablet_manager.cc:1905] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:27.173099 8723 ts_tablet_manager.cc:1918] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:27.173189 8723 log.cc:1196] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/8beb59cb2249425b9eb46b6260e6e9a8
I20260927 07:08:27.173445 8723 ts_tablet_manager.cc:1939] T 8beb59cb2249425b9eb46b6260e6e9a8 P 1da71ce482a7418e9028686305693d92: Deleting consensus metadata
I20260927 07:08:27.173854 8208 catalog_manager.cc:4841] TS 1da71ce482a7418e9028686305693d92 (127.21.2.66:37595): tablet 8beb59cb2249425b9eb46b6260e6e9a8 (table test-table [id=ef7132a741d14cd7b0464785c550caca]) successfully deleted
I20260927 07:08:27.176009 8724 tablet_replica.cc:330] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: stopping tablet replica
I20260927 07:08:27.176155 8724 raft_consensus.cc:2229] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:27.176326 8724 raft_consensus.cc:2258] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:08:27.176376 8211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:41025 (error 108)
I20260927 07:08:27.176652 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8420
I20260927 07:08:27.177369 8724 ts_tablet_manager.cc:1905] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:27.178854 8724 ts_tablet_manager.cc:1918] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:27.178941 8724 log.cc:1196] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/8beb59cb2249425b9eb46b6260e6e9a8
I20260927 07:08:27.179275 8724 ts_tablet_manager.cc:1939] T 8beb59cb2249425b9eb46b6260e6e9a8 P b5f0992da98446d0ae437c293bafcc9d: Deleting consensus metadata
I20260927 07:08:27.179706 8210 catalog_manager.cc:4841] TS b5f0992da98446d0ae437c293bafcc9d (127.21.2.67:45869): tablet 8beb59cb2249425b9eb46b6260e6e9a8 (table test-table [id=ef7132a741d14cd7b0464785c550caca]) successfully deleted
I20260927 07:08:27.182089 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.177698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:41025 (local address 127.0.0.1:34196)
0927 07:08:27.181998 (+ 4300us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.182051 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4272}
W20260927 07:08:27.182277 8211 catalog_manager.cc:4584] TS 0eff13f250e64fb993340607c8a98b25 (127.21.2.65:41025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8beb59cb2249425b9eb46b6260e6e9a8: Network error: Client connection negotiation failed: client connection to 127.21.2.65:41025: connect: Connection refused (error 111)
I20260927 07:08:27.183153 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8554
I20260927 07:08:27.188330 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8192
2026-09-27T07:08:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2086 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T07:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:27Z Disabled control of system clock
I20260927 07:08:27.217900 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34829
--webserver_interface=127.21.2.126
--webserver_port=0
--builtin_ntp_servers=127.21.2.84:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:27.301229 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:27.301553 8730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:27.301589 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:27.303349 8730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:27.303440 8730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:27.303463 8730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:27.303494 8730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:27.303520 8730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:27.304920 8730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:27.305207 8730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:27.305495 8730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:27.310289 8736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:27.310379 8730 server_base.cc:1034] running on GCE node
W20260927 07:08:27.310289 8735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.310328 8738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:27.310719 8730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:27.311038 8730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:27.312191 8730 hybrid_clock.cc:648] HybridClock initialized: now 1790492907312190 us; error 36 us; skew 500 ppm
I20260927 07:08:27.313390 8730 webserver.cc:460] Webserver started at http://127.21.2.126:39753/ using document root <none> and password file <none>
I20260927 07:08:27.313632 8730 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:27.313699 8730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:27.313864 8730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:27.314872 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "b23f3c4393a04bda8680e56f595d47fd"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "ec5eb14870a9425b2bf6a236eace67b2"
server_key_iv: "570957fc9e0b7adf0d1815174ebef5a8"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.315261 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "b23f3c4393a04bda8680e56f595d47fd"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "ec5eb14870a9425b2bf6a236eace67b2"
server_key_iv: "570957fc9e0b7adf0d1815174ebef5a8"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.316550 8730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:27.317251 8744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.317410 8730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:27.317503 8730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "b23f3c4393a04bda8680e56f595d47fd"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "ec5eb14870a9425b2bf6a236eace67b2"
server_key_iv: "570957fc9e0b7adf0d1815174ebef5a8"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.317587 8730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:27.334620 8730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:27.334970 8730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:27.335129 8730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:27.339053 8730 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34829
I20260927 07:08:27.339084 8796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34829 every 8 connection(s)
I20260927 07:08:27.339460 8730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:27.340364 8797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:27.342543 8797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: Bootstrap starting.
I20260927 07:08:27.342703 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8730
I20260927 07:08:27.342840 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:27.343043 21513 external_mini_cluster.cc:1428] Setting key c6749b625a83687101dc881cc0e44d98
I20260927 07:08:27.343312 8797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:27.343762 8797 log.cc:824] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:27.344501 8797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: No bootstrap required, opened a new log
I20260927 07:08:27.348093 8797 raft_consensus.cc:348] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } }
I20260927 07:08:27.348294 8797 raft_consensus.cc:374] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:27.348332 8797 raft_consensus.cc:729] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23f3c4393a04bda8680e56f595d47fd, State: Initialized, Role: FOLLOWER
I20260927 07:08:27.348459 8797 consensus_queue.cc:260] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } }
I20260927 07:08:27.348541 8797 raft_consensus.cc:388] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:27.348573 8797 raft_consensus.cc:482] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:27.348619 8797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.349418 8797 raft_consensus.cc:504] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } }
I20260927 07:08:27.349545 8797 leader_election.cc:302] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b23f3c4393a04bda8680e56f595d47fd; no voters:
I20260927 07:08:27.349733 8797 leader_election.cc:288] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:27.349766 8802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:27.349870 8802 raft_consensus.cc:686] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [term 1 LEADER]: Becoming Leader. State: Replica: b23f3c4393a04bda8680e56f595d47fd, State: Running, Role: LEADER
I20260927 07:08:27.349967 8802 consensus_queue.cc:237] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } }
I20260927 07:08:27.350054 8797 sys_catalog.cc:564] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:27.350289 8802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b23f3c4393a04bda8680e56f595d47fd. Latest consensus state: current_term: 1 leader_uuid: "b23f3c4393a04bda8680e56f595d47fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } } }
I20260927 07:08:27.350361 8802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:27.350304 8803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b23f3c4393a04bda8680e56f595d47fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23f3c4393a04bda8680e56f595d47fd" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34829 } } }
I20260927 07:08:27.350533 8803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:27.350571 8806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:27.350610 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.343584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.21.2.126:34829)
0927 07:08:27.343806 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.343815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.343849 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:08:27.344424 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.344433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.344507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.345341 (+ 834us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:27.345575 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.347400 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.348239 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.348849 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.348988 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.349654 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:27.349659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:27.349659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:27.349900 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:27.349946 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:27.349963 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:27.350270 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.350304 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.350390 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260927 07:08:27.351095 8806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:27.352737 8806 catalog_manager.cc:1347] Generated new cluster ID: 09e2dbb4fdeb490aa6055a0ca56cf747
I20260927 07:08:27.352792 8806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:27.371585 8806 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:27.372113 8806 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:27.377694 8806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: Generated new TSK 0
I20260927 07:08:27.377882 8806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:27.379979 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.65:0
--local_ip_for_outbound_sockets=127.21.2.65
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--builtin_ntp_servers=127.21.2.84:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:08:27.461385 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:27.461593 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:27.461615 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:27.462786 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:27.462829 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:27.462855 8821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.65
I20260927 07:08:27.464231 8821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.2.65
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.65
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:27.464464 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:27.464807 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:27.469779 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.469792 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.469807 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:27.470175 8821 server_base.cc:1034] running on GCE node
I20260927 07:08:27.470404 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:27.470676 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:27.471822 8821 hybrid_clock.cc:648] HybridClock initialized: now 1790492907471811 us; error 28 us; skew 500 ppm
I20260927 07:08:27.473148 8821 webserver.cc:460] Webserver started at http://127.21.2.65:37977/ using document root <none> and password file <none>
I20260927 07:08:27.473398 8821 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:27.473443 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:27.473582 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:27.474644 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data/instance:
uuid: "475d8da5c74048ef94b38013b2f96c24"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2a59b3facdbdb93d2a99bfbf35e87760"
server_key_iv: "aa5ddbca80d1a83ce035609a368eb32f"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.475001 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance:
uuid: "475d8da5c74048ef94b38013b2f96c24"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2a59b3facdbdb93d2a99bfbf35e87760"
server_key_iv: "aa5ddbca80d1a83ce035609a368eb32f"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.476393 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:27.477159 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.477321 8821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:27.477403 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal
uuid: "475d8da5c74048ef94b38013b2f96c24"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2a59b3facdbdb93d2a99bfbf35e87760"
server_key_iv: "aa5ddbca80d1a83ce035609a368eb32f"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.477470 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:27.486332 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:27.486601 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:27.486744 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:27.486958 8821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:27.487275 8821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:27.487330 8821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.487391 8821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:27.487419 8821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.493265 8821 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.65:37325
I20260927 07:08:27.493302 8948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.65:37325 every 8 connection(s)
I20260927 07:08:27.493642 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:27.495026 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8821
I20260927 07:08:27.495093 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:27.495306 21513 external_mini_cluster.cc:1428] Setting key 007399d0e797931700b395951fc25d4a
I20260927 07:08:27.496352 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.66:0
--local_ip_for_outbound_sockets=127.21.2.66
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--builtin_ntp_servers=127.21.2.84:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:08:27.499197 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.494393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:37141 (local address 127.21.2.126:34829)
0927 07:08:27.494446 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.494448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.494702 (+ 254us) server_negotiation.cc:407] Connection header received
0927 07:08:27.495235 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.495236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.495248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.495299 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:27.496216 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.496799 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.497768 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.497961 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.498037 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.498901 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:27.498902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:27.498903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:27.498907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:27.498921 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:27.498925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:27.498982 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.499098 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.499122 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:08:27.499449 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.494385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34829 (local address 127.21.2.65:37141)
0927 07:08:27.494583 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.494616 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.495112 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.495407 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.495411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.495421 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:27.496128 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.496135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.496902 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.496903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.497667 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.497669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.498052 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.498053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.498188 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.498745 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:27.498746 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:27.498773 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:27.498795 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:27.498993 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:27.498998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:27.499001 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.499098 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.499197 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260927 07:08:27.499917 8949 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34829
I20260927 07:08:27.500025 8949 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:27.500236 8949 heartbeater.cc:511] Master 127.21.2.126:34829 requested a full tablet report, sending...
I20260927 07:08:27.500721 8761 ts_manager.cc:194] Registered new tserver with Master: 475d8da5c74048ef94b38013b2f96c24 (127.21.2.65:37325)
I20260927 07:08:27.501322 8761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.65:37141
W20260927 07:08:27.579121 8952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:27.579329 8952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:27.579351 8952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:27.580642 8952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:27.580686 8952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:27.580711 8952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.66
I20260927 07:08:27.582134 8952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.2.66
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.66
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:27.582379 8952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:27.582682 8952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:27.587630 8952 server_base.cc:1034] running on GCE node
W20260927 07:08:27.587598 8957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.587590 8958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.587576 8960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:27.587981 8952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:27.588301 8952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:27.589470 8952 hybrid_clock.cc:648] HybridClock initialized: now 1790492907589447 us; error 37 us; skew 500 ppm
I20260927 07:08:27.590764 8952 webserver.cc:460] Webserver started at http://127.21.2.66:40051/ using document root <none> and password file <none>
I20260927 07:08:27.591032 8952 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:27.591077 8952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:27.591163 8952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:27.592232 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data/instance:
uuid: "8f1e2be523b24a8bb08d836ed6afc736"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "76ba5f7c22c0357c4307fa715f463afd"
server_key_iv: "31411cc6fc22bbff75f6a0a238cfb4a7"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.592532 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1e2be523b24a8bb08d836ed6afc736"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "76ba5f7c22c0357c4307fa715f463afd"
server_key_iv: "31411cc6fc22bbff75f6a0a238cfb4a7"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.593756 8952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:08:27.594406 8966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.594604 8952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:27.594686 8952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal
uuid: "8f1e2be523b24a8bb08d836ed6afc736"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "76ba5f7c22c0357c4307fa715f463afd"
server_key_iv: "31411cc6fc22bbff75f6a0a238cfb4a7"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.594740 8952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:27.613221 8952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:27.613494 8952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:27.613595 8952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:27.613780 8952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:27.614138 8952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:27.614177 8952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.614200 8952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:27.614250 8952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.619935 8952 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.66:37529
I20260927 07:08:27.620021 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.66:37529 every 8 connection(s)
I20260927 07:08:27.620381 8952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:27.621411 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 8952
I20260927 07:08:27.621529 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:27.621727 21513 external_mini_cluster.cc:1428] Setting key 5c90755608ea1f56692dd05b756c10d7
I20260927 07:08:27.622653 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.2.67:0
--local_ip_for_outbound_sockets=127.21.2.67
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--builtin_ntp_servers=127.21.2.84:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:08:27.625988 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.621268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.66:38179 (local address 127.21.2.126:34829)
0927 07:08:27.621340 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.621342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.621552 (+ 210us) server_negotiation.cc:407] Connection header received
0927 07:08:27.622152 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.622153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.622166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.622217 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:27.622927 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.623508 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.624502 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.624730 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.624809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.625696 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:27.625698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:27.625698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:27.625703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:27.625718 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:27.625723 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:27.625772 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.625879 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.625906 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:08:27.626216 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.621235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34829 (local address 127.21.2.66:38179)
0927 07:08:27.621413 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.621454 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.622038 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.622283 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.622287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.622296 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:27.622853 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.622859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.623606 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.623607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.624405 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.624407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.624816 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.624817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.624960 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.625541 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:27.625542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:27.625570 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:27.625593 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:27.625782 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:27.625787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:27.625791 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.625880 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.625982 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260927 07:08:27.626698 9080 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34829
I20260927 07:08:27.626816 9080 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:27.627043 9080 heartbeater.cc:511] Master 127.21.2.126:34829 requested a full tablet report, sending...
I20260927 07:08:27.627441 8761 ts_manager.cc:194] Registered new tserver with Master: 8f1e2be523b24a8bb08d836ed6afc736 (127.21.2.66:37529)
I20260927 07:08:27.627820 8761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.66:38179
W20260927 07:08:27.705505 9083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:27.705716 9083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:27.705739 9083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:27.706998 9083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:27.707038 9083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:27.707064 9083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.2.67
I20260927 07:08:27.708461 9083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.2.67
--webserver_port=0
--tserver_master_addrs=127.21.2.126:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.2.67
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:27.708736 9083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:27.709043 9083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:27.714069 9091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.714078 9089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:27.714107 9088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:27.714107 9083 server_base.cc:1034] running on GCE node
I20260927 07:08:27.714427 9083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:27.714665 9083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:27.715802 9083 hybrid_clock.cc:648] HybridClock initialized: now 1790492907715781 us; error 38 us; skew 500 ppm
I20260927 07:08:27.717039 9083 webserver.cc:460] Webserver started at http://127.21.2.67:33121/ using document root <none> and password file <none>
I20260927 07:08:27.717316 9083 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:27.717363 9083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:27.717531 9083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:27.718573 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data/instance:
uuid: "51e92063681c4f82831479c6d12d201d"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2633182770f3d35622478a1b1f1c65ed"
server_key_iv: "1ce26a31e24b1758c50134ea740d003c"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.718919 9083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance:
uuid: "51e92063681c4f82831479c6d12d201d"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2633182770f3d35622478a1b1f1c65ed"
server_key_iv: "1ce26a31e24b1758c50134ea740d003c"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.720346 9083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:27.721086 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.721302 9083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:27.721395 9083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal
uuid: "51e92063681c4f82831479c6d12d201d"
format_stamp: "Formatted at 2026-09-27 07:08:27 on dist-test-slave-ltqz"
server_key: "2633182770f3d35622478a1b1f1c65ed"
server_key_iv: "1ce26a31e24b1758c50134ea740d003c"
server_key_version: "encryptionkey@0"
I20260927 07:08:27.721469 9083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:27.759758 9083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:27.760094 9083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:27.760242 9083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:27.760506 9083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:27.760846 9083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:27.760879 9083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.760902 9083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:27.760915 9083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:27.766561 9083 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.67:37473
I20260927 07:08:27.766630 9210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.67:37473 every 8 connection(s)
I20260927 07:08:27.766923 9083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:27.767907 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9083
I20260927 07:08:27.768016 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:27.768211 21513 external_mini_cluster.cc:1428] Setting key 0c19320d5ad9f97c086da03135364fc7
I20260927 07:08:27.772354 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.767683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.67:36159 (local address 127.21.2.126:34829)
0927 07:08:27.767753 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.767755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.768050 (+ 295us) server_negotiation.cc:407] Connection header received
0927 07:08:27.768688 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.768689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.768702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.768752 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:27.769539 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.770017 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.771030 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.771201 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.771249 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.772152 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:27.772153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:27.772154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:27.772157 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:27.772169 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:27.772172 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:27.772212 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.772280 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.772298 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:08:27.772593 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.767681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34829 (local address 127.21.2.67:36159)
0927 07:08:27.767897 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.767936 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.768584 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.768839 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.768844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.768854 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:27.769460 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.769465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.770136 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.770137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.770950 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.770952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.771280 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.771280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.771405 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.772020 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:08:27.772021 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:27.772056 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:08:27.772078 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:27.772215 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:27.772217 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:27.772219 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.772275 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.772352 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260927 07:08:27.773072 9211 heartbeater.cc:347] Connected to a master server at 127.21.2.126:34829
I20260927 07:08:27.773160 9211 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:27.773298 9211 heartbeater.cc:511] Master 127.21.2.126:34829 requested a full tablet report, sending...
I20260927 07:08:27.773631 8761 ts_manager.cc:194] Registered new tserver with Master: 51e92063681c4f82831479c6d12d201d (127.21.2.67:37473)
I20260927 07:08:27.774021 8761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.2.67:36159
I20260927 07:08:27.779922 21513 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:27.784220 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.781324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.21.2.126:34829)
0927 07:08:27.781417 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.781418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.781458 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:08:27.781520 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.781520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.781528 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.781567 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:27.781716 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.782177 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.782967 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.783146 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.783193 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.783949 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:27.783949 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:27.783950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:27.783954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:27.783964 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:27.783968 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:27.784008 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.784117 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.784136 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:08:27.785928 8761 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:27.787457 8761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:27.793865 9220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.789741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37325 (local address 127.0.0.1:57892)
0927 07:08:27.790051 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.790093 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.790162 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.790402 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.790405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.790407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.790466 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.790468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.791523 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.791524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.792661 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.792664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.793555 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.793635 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.793648 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.793728 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.793782 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260927 07:08:27.793886 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.789844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.21.2.65:37325)
0927 07:08:27.790152 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.790156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.790166 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:08:27.790287 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.790290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.790305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.790382 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.790568 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.791428 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.792756 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.793447 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793551 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.793647 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.793743 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.793782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
I20260927 07:08:27.794241 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.789742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37529 (local address 127.0.0.1:55226)
0927 07:08:27.790103 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.790114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.790175 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.790513 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.790514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.790517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.790576 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.790578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.791976 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.791977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.792959 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.792961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.793972 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.794036 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.794040 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.794111 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.794160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260927 07:08:27.794310 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.790035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.21.2.67:37473)
0927 07:08:27.790252 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.790256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.790592 (+ 336us) server_negotiation.cc:407] Connection header received
0927 07:08:27.790639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.790642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.790659 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.790747 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.790860 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.791953 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793173 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.793869 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793928 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.793999 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.794172 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.794218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260927 07:08:27.794240 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.789959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:37473 (local address 127.0.0.1:50920)
0927 07:08:27.790218 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.790228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.790270 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.790760 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.790761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.790763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.790796 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.790797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.792033 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.792034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.793079 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.793082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.794027 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.794028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.794094 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.794097 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.794150 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.794179 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":79,"threads_started":1}
I20260927 07:08:27.794574 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.789844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.21.2.66:37529)
0927 07:08:27.790057 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.790062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.790269 (+ 207us) server_negotiation.cc:407] Connection header received
0927 07:08:27.790322 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.790324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.790342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.790438 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.790666 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.791881 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793044 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.793776 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.793849 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.793934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.794435 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.794478 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":104,"threads_started":1}
I20260927 07:08:27.794989 8883 tablet_service.cc:1467] Processing CreateTablet for tablet 70aab52f60f6477eb640d39922aca576 (DEFAULT_TABLE table=test-table [id=6edf4aad0fe545e2b03c341502fbf6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:27.795190 9014 tablet_service.cc:1467] Processing CreateTablet for tablet 70aab52f60f6477eb640d39922aca576 (DEFAULT_TABLE table=test-table [id=6edf4aad0fe545e2b03c341502fbf6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:27.795440 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aab52f60f6477eb640d39922aca576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:27.795589 9014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aab52f60f6477eb640d39922aca576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:27.798349 9225 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Bootstrap starting.
I20260927 07:08:27.798661 9226 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: Bootstrap starting.
I20260927 07:08:27.798852 9145 tablet_service.cc:1467] Processing CreateTablet for tablet 70aab52f60f6477eb640d39922aca576 (DEFAULT_TABLE table=test-table [id=6edf4aad0fe545e2b03c341502fbf6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:27.799170 9145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70aab52f60f6477eb640d39922aca576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:27.799228 9225 tablet_bootstrap.cc:654] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:27.799279 9226 tablet_bootstrap.cc:654] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:27.799628 9226 log.cc:824] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:27.799628 9225 log.cc:824] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:27.800397 9225 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: No bootstrap required, opened a new log
I20260927 07:08:27.800474 9225 ts_tablet_manager.cc:1397] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:08:27.800640 9226 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: No bootstrap required, opened a new log
I20260927 07:08:27.800709 9226 ts_tablet_manager.cc:1397] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:27.802124 9229 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Bootstrap starting.
I20260927 07:08:27.802678 9229 tablet_bootstrap.cc:654] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:27.802974 9229 log.cc:824] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:27.803319 9226 raft_consensus.cc:348] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.803319 9225 raft_consensus.cc:348] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.803524 9225 raft_consensus.cc:374] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:27.803524 9226 raft_consensus.cc:374] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:27.803566 9225 raft_consensus.cc:729] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1e2be523b24a8bb08d836ed6afc736, State: Initialized, Role: FOLLOWER
I20260927 07:08:27.803566 9226 raft_consensus.cc:729] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475d8da5c74048ef94b38013b2f96c24, State: Initialized, Role: FOLLOWER
I20260927 07:08:27.803721 9229 tablet_bootstrap.cc:492] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: No bootstrap required, opened a new log
I20260927 07:08:27.803711 9226 consensus_queue.cc:260] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.803711 9225 consensus_queue.cc:260] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.803786 9229 ts_tablet_manager.cc:1397] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:27.803989 9226 ts_tablet_manager.cc:1428] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 07:08:27.803989 9225 ts_tablet_manager.cc:1428] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 07:08:27.804028 8949 heartbeater.cc:503] Master 127.21.2.126:34829 was elected leader, sending a full tablet report...
I20260927 07:08:27.804112 9080 heartbeater.cc:503] Master 127.21.2.126:34829 was elected leader, sending a full tablet report...
I20260927 07:08:27.806152 9229 raft_consensus.cc:348] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.806296 9229 raft_consensus.cc:374] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:27.806342 9229 raft_consensus.cc:729] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e92063681c4f82831479c6d12d201d, State: Initialized, Role: FOLLOWER
I20260927 07:08:27.806449 9229 consensus_queue.cc:260] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.806699 9229 ts_tablet_manager.cc:1428] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:27.806787 9211 heartbeater.cc:503] Master 127.21.2.126:34829 was elected leader, sending a full tablet report...
W20260927 07:08:27.871114 9081 tablet.cc:2369] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:27.960423 9231 raft_consensus.cc:482] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:27.960557 9231 raft_consensus.cc:504] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.960965 9231 leader_election.cc:288] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51e92063681c4f82831479c6d12d201d (127.21.2.67:37473), 8f1e2be523b24a8bb08d836ed6afc736 (127.21.2.66:37529)
I20260927 07:08:27.963996 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.961172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.67:37473 (local address 127.21.2.65:48203)
0927 07:08:27.961219 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.961230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.961297 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.961519 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.961520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.961522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.961597 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.961600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.962321 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.962322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.963118 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.963120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.963738 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.963813 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.963816 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.963872 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.963912 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260927 07:08:27.964012 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.961316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:48203 (local address 127.21.2.67:37473)
0927 07:08:27.961380 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.961381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.961389 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:27.961416 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.961417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.961424 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.961512 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.961670 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.962234 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963207 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.963655 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963710 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.963719 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.963888 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.963930 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:27.964100 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.961172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.66:37529 (local address 127.21.2.65:47497)
0927 07:08:27.961513 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.961526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:27.961574 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:27.961736 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:27.961737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:27.961740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:27.961811 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.961812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.962448 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.962449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.963206 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:27.963208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.963820 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:27.963918 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.963921 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:08:27.963979 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:08:27.964015 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":174,"thread_start_us":117,"threads_started":1}
I20260927 07:08:27.964126 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.961244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.2.65:47497 (local address 127.21.2.66:37529)
0927 07:08:27.961295 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.961297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.961575 (+ 278us) server_negotiation.cc:407] Connection header received
0927 07:08:27.961641 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.961643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.961657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.961715 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:27.961883 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.962365 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963295 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.963734 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.963808 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.963817 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.964002 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.964049 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:08:27.964375 9034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aab52f60f6477eb640d39922aca576" candidate_uuid: "475d8da5c74048ef94b38013b2f96c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1e2be523b24a8bb08d836ed6afc736" is_pre_election: true
I20260927 07:08:27.964382 9165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aab52f60f6477eb640d39922aca576" candidate_uuid: "475d8da5c74048ef94b38013b2f96c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e92063681c4f82831479c6d12d201d" is_pre_election: true
I20260927 07:08:27.964527 9034 raft_consensus.cc:2454] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 475d8da5c74048ef94b38013b2f96c24 in term 0.
I20260927 07:08:27.964527 9165 raft_consensus.cc:2454] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 475d8da5c74048ef94b38013b2f96c24 in term 0.
I20260927 07:08:27.964792 8836 leader_election.cc:302] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475d8da5c74048ef94b38013b2f96c24, 8f1e2be523b24a8bb08d836ed6afc736; no voters:
I20260927 07:08:27.964946 9231 raft_consensus.cc:2781] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:27.965004 9231 raft_consensus.cc:482] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:27.965032 9231 raft_consensus.cc:3037] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.965721 9231 raft_consensus.cc:504] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.965888 9231 leader_election.cc:288] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [CANDIDATE]: Term 1 election: Requested vote from peers 51e92063681c4f82831479c6d12d201d (127.21.2.67:37473), 8f1e2be523b24a8bb08d836ed6afc736 (127.21.2.66:37529)
I20260927 07:08:27.966063 9034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aab52f60f6477eb640d39922aca576" candidate_uuid: "475d8da5c74048ef94b38013b2f96c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1e2be523b24a8bb08d836ed6afc736"
I20260927 07:08:27.966054 9165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70aab52f60f6477eb640d39922aca576" candidate_uuid: "475d8da5c74048ef94b38013b2f96c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51e92063681c4f82831479c6d12d201d"
I20260927 07:08:27.966137 9034 raft_consensus.cc:3037] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.966157 9165 raft_consensus.cc:3037] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:27.966817 9034 raft_consensus.cc:2454] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475d8da5c74048ef94b38013b2f96c24 in term 1.
I20260927 07:08:27.966910 9165 raft_consensus.cc:2454] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475d8da5c74048ef94b38013b2f96c24 in term 1.
I20260927 07:08:27.966979 8836 leader_election.cc:302] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475d8da5c74048ef94b38013b2f96c24, 8f1e2be523b24a8bb08d836ed6afc736; no voters:
I20260927 07:08:27.967067 9231 raft_consensus.cc:2781] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:27.967217 9231 raft_consensus.cc:686] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 1 LEADER]: Becoming Leader. State: Replica: 475d8da5c74048ef94b38013b2f96c24, State: Running, Role: LEADER
I20260927 07:08:27.967319 9231 consensus_queue.cc:237] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } }
I20260927 07:08:27.968338 8760 catalog_manager.cc:5495] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 475d8da5c74048ef94b38013b2f96c24 (127.21.2.65). New cstate: current_term: 1 leader_uuid: "475d8da5c74048ef94b38013b2f96c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475d8da5c74048ef94b38013b2f96c24" member_type: VOTER last_known_addr { host: "127.21.2.65" port: 37325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 } health_report { overall_health: UNKNOWN } } }
I20260927 07:08:27.973948 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.971902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.21.2.65:37325)
0927 07:08:27.971957 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:27.971959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:27.971966 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:08:27.972032 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:27.972033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:27.972048 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:27.972106 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:27.972202 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.972637 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.973306 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:27.973479 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:27.973527 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:27.973650 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:27.973733 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:27.973785 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:27.973821 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 07:08:27.973836 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:08:27.975708 9165 raft_consensus.cc:1261] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 1 FOLLOWER]: Refusing update from remote peer 475d8da5c74048ef94b38013b2f96c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:27.975708 9034 raft_consensus.cc:1261] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 1 FOLLOWER]: Refusing update from remote peer 475d8da5c74048ef94b38013b2f96c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:27.976037 9231 consensus_queue.cc:1035] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51e92063681c4f82831479c6d12d201d" member_type: VOTER last_known_addr { host: "127.21.2.67" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:27.976212 9235 consensus_queue.cc:1035] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1e2be523b24a8bb08d836ed6afc736" member_type: VOTER last_known_addr { host: "127.21.2.66" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:27.979236 9239 mvcc.cc:205] Tried to move back new op lower bound from 7333858951065661440 to 7333858951035211776. Current Snapshot: MvccSnapshot[applied={T|T < 7333858951065661440 or (T in {7333858951065661440})}]
I20260927 07:08:27.980430 8761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:27.980520 8761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:27.981318 8761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b23f3c4393a04bda8680e56f595d47fd: Sending DeleteTablet for 3 replicas of tablet 70aab52f60f6477eb640d39922aca576
I20260927 07:08:27.981670 9145 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aab52f60f6477eb640d39922aca576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='slave'} at 127.0.0.1:50920
I20260927 07:08:27.981668 8883 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aab52f60f6477eb640d39922aca576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='slave'} at 127.0.0.1:57892
I20260927 07:08:27.981880 9248 tablet_replica.cc:330] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24: stopping tablet replica
I20260927 07:08:27.981971 9248 raft_consensus.cc:2229] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:27.982095 9248 raft_consensus.cc:2258] T 70aab52f60f6477eb640d39922aca576 P 475d8da5c74048ef94b38013b2f96c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:27.982173 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8821
I20260927 07:08:27.982215 9014 tablet_service.cc:1514] Processing DeleteTablet for tablet 70aab52f60f6477eb640d39922aca576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:27 UTC) from {username='slave'} at 127.0.0.1:55226
I20260927 07:08:27.983289 9250 tablet_replica.cc:330] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: stopping tablet replica
I20260927 07:08:27.983409 9250 raft_consensus.cc:2229] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:27.983508 9250 raft_consensus.cc:2258] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:27.983942 9250 ts_tablet_manager.cc:1905] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:27.985919 9250 ts_tablet_manager.cc:1918] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:27.985982 9250 log.cc:1196] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-1/wal/wals/70aab52f60f6477eb640d39922aca576
I20260927 07:08:27.986225 9250 ts_tablet_manager.cc:1939] T 70aab52f60f6477eb640d39922aca576 P 8f1e2be523b24a8bb08d836ed6afc736: Deleting consensus metadata
I20260927 07:08:27.986734 8745 catalog_manager.cc:4841] TS 8f1e2be523b24a8bb08d836ed6afc736 (127.21.2.66:37529): tablet 70aab52f60f6477eb640d39922aca576 (table test-table [id=6edf4aad0fe545e2b03c341502fbf6f0]) successfully deleted
I20260927 07:08:27.986953 9249 tablet_replica.cc:330] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: stopping tablet replica
I20260927 07:08:27.987072 9249 raft_consensus.cc:2229] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:27.987190 9249 raft_consensus.cc:2258] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:27.987592 9249 ts_tablet_manager.cc:1905] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 07:08:27.988193 8747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.2.65:37325 (error 108)
I20260927 07:08:27.988426 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8952
I20260927 07:08:27.988631 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:27.988512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.65:37325 (local address 127.0.0.1:57904)
0927 07:08:27.988559 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:27.988594 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:08:27.988735 8747 catalog_manager.cc:4584] TS 475d8da5c74048ef94b38013b2f96c24 (127.21.2.65:37325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70aab52f60f6477eb640d39922aca576: Network error: Client connection negotiation failed: client connection to 127.21.2.65:37325: connect: Connection refused (error 111)
I20260927 07:08:27.991012 9249 ts_tablet_manager.cc:1918] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:27.991101 9249 log.cc:1196] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Deleting WAL directory at /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492641704908-21513-0/minicluster-data/ts-2/wal/wals/70aab52f60f6477eb640d39922aca576
I20260927 07:08:27.991319 9249 ts_tablet_manager.cc:1939] T 70aab52f60f6477eb640d39922aca576 P 51e92063681c4f82831479c6d12d201d: Deleting consensus metadata
I20260927 07:08:27.991743 8747 catalog_manager.cc:4841] TS 51e92063681c4f82831479c6d12d201d (127.21.2.67:37473): tablet 70aab52f60f6477eb640d39922aca576 (table test-table [id=6edf4aad0fe545e2b03c341502fbf6f0]) successfully deleted
I20260927 07:08:27.992923 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9083
I20260927 07:08:27.997900 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 8730
2026-09-27T07:08:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (808 ms)
[----------] 6 tests from EncryptionPolicyTest (8324 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:28 dist-test-slave-ltqz krb5kdc[9254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:28 dist-test-slave-ltqz krb5kdc[9254](info): set up 2 sockets
Sep 27 07:08:28 dist-test-slave-ltqz krb5kdc[9254](info): commencing operation
krb5kdc: starting...
W20260927 07:08:29.039528 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:29.154637 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:29.240789 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:29.241022 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:29.241072 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:29.242233 9267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:29.242300 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:29.242326 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:29.242352 9267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:29.242383 9267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:29.243857 9267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:29.244132 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:29.244361 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:29.249603 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:29.249614 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:29.249614 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:29.249850 9267 server_base.cc:1034] running on GCE node
I20260927 07:08:29.250111 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:29.250432 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:29.251590 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790492909251573 us; error 40 us; skew 500 ppm
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:29.254009 9267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:29.254419 9267 webserver.cc:460] Webserver started at http://127.0.0.1:42803/ using document root <none> and password file <none>
I20260927 07:08:29.254667 9267 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:29.254738 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:29.254882 9267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:29.255981 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "be5a5e387a7f45c38d90f81855d4f69a"
format_stamp: "Formatted at 2026-09-27 07:08:29 on dist-test-slave-ltqz"
server_key: "ba27d7dc3bd79b7d4e5be461cc6338fe"
server_key_iv: "cd5a48682cd95f00bcdb85898d4f7d32"
server_key_version: "encryptionkey@0"
I20260927 07:08:29.256330 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "be5a5e387a7f45c38d90f81855d4f69a"
format_stamp: "Formatted at 2026-09-27 07:08:29 on dist-test-slave-ltqz"
server_key: "ba27d7dc3bd79b7d4e5be461cc6338fe"
server_key_iv: "cd5a48682cd95f00bcdb85898d4f7d32"
server_key_version: "encryptionkey@0"
I20260927 07:08:29.257601 9267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:29.258261 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:29.258425 9267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:29.258538 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "be5a5e387a7f45c38d90f81855d4f69a"
format_stamp: "Formatted at 2026-09-27 07:08:29 on dist-test-slave-ltqz"
server_key: "ba27d7dc3bd79b7d4e5be461cc6338fe"
server_key_iv: "cd5a48682cd95f00bcdb85898d4f7d32"
server_key_version: "encryptionkey@0"
I20260927 07:08:29.258612 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:29.297472 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:29.299968 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:29.300137 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:29.304046 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35061
I20260927 07:08:29.304086 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35061 every 8 connection(s)
I20260927 07:08:29.304456 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:29.305317 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:29.307933 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: Bootstrap starting.
I20260927 07:08:29.308650 9335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:29.308998 9335 log.cc:824] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:29.309898 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: No bootstrap required, opened a new log
I20260927 07:08:29.310091 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9267
I20260927 07:08:29.310247 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:29.310438 21513 external_mini_cluster.cc:1428] Setting key 900dfdf611fdb1576471ce4be64912d4
I20260927 07:08:29.312392 9335 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260927 07:08:29.312599 9335 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:29.312641 9335 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5a5e387a7f45c38d90f81855d4f69a, State: Initialized, Role: FOLLOWER
I20260927 07:08:29.312788 9335 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260927 07:08:29.312882 9335 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:29.312917 9335 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:29.312963 9335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:29.313558 9335 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260927 07:08:29.313645 9335 leader_election.cc:302] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5a5e387a7f45c38d90f81855d4f69a; no voters:
I20260927 07:08:29.313824 9335 leader_election.cc:288] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:29.313846 9340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:29.313958 9340 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [term 1 LEADER]: Becoming Leader. State: Replica: be5a5e387a7f45c38d90f81855d4f69a, State: Running, Role: LEADER
I20260927 07:08:29.314052 9340 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } }
I20260927 07:08:29.314091 9335 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:29.314386 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5a5e387a7f45c38d90f81855d4f69a. Latest consensus state: current_term: 1 leader_uuid: "be5a5e387a7f45c38d90f81855d4f69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260927 07:08:29.314468 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:29.314395 9341 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5a5e387a7f45c38d90f81855d4f69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5a5e387a7f45c38d90f81855d4f69a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35061 } } }
I20260927 07:08:29.314656 9341 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a [sys.catalog]: This master's current role is: LEADER
W20260927 07:08:29.315861 9355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:29.315943 9355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:29.316155 9345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:29.316358 9345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:08:29.320497 9345 catalog_manager.cc:1347] Generated new cluster ID: c695094da0e24b33b0bfa41e2af2062d
I20260927 07:08:29.320555 9345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:29.322695 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:29.311020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56889 (local address 127.0.0.1:35061)
0927 07:08:29.311239 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:29.311247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:29.311302 (+ 55us) server_negotiation.cc:407] Connection header received
0927 07:08:29.311928 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:29.311935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:29.311981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:29.312096 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:29.313054 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:29.314523 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:29.315337 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:29.316054 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:29.316197 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:29.319073 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:29.319078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:29.319078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:29.319090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:29.321069 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:29.321465 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:29.321468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:29.321468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:29.321565 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:29.321795 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:29.321796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:29.321796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:29.322045 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:29.322213 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:29.322332 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 07:08:29.322537 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260927 07:08:29.324296 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:29.333645 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:29.325686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54985 (local address 127.0.0.1:35061)
0927 07:08:29.325724 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:29.325726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:29.327546 (+ 1820us) server_negotiation.cc:407] Connection header received
0927 07:08:29.327621 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:29.327622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:29.327633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:29.327692 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:29.328212 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:29.328725 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:29.329452 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:29.329686 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:29.329770 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:29.330993 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:29.330995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:29.330995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:29.331002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:29.332465 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:29.332847 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:29.332849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:29.332849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:29.332909 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:29.333131 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:29.333132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:29.333133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:29.333249 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:29.333372 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:29.333475 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 07:08:29.333537 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:08:29.339119 9345 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:29.339887 9345 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:29.350983 9345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5a5e387a7f45c38d90f81855d4f69a: Generated new TSK 0
I20260927 07:08:29.351210 9345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:29.353631 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9267
W20260927 07:08:29.358098 9252 connection.cc:535] client connection to 127.0.0.1:35061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35061): Connection reset by peer (error 104)
2026-09-27T07:08:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9366](info): set up 2 sockets
Sep 27 07:08:29 dist-test-slave-ltqz krb5kdc[9366](info): commencing operation
krb5kdc: starting...
W20260927 07:08:30.387290 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:30 dist-test-slave-ltqz krb5kdc[9366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:30.504005 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:30.584050 9379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:30.584246 9379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:30.584268 9379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:30.585381 9379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:30.585415 9379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:30.585429 9379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:30.585443 9379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:30.585454 9379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:30.586807 9379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:30.586995 9379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:30.587157 9379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:30.591675 9385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:30.591667 9384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:30.591666 9387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:30.592031 9379 server_base.cc:1034] running on GCE node
I20260927 07:08:30.592262 9379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:30.592532 9379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:30.593693 9379 hybrid_clock.cc:648] HybridClock initialized: now 1790492910593686 us; error 35 us; skew 500 ppm
Sep 27 07:08:30 dist-test-slave-ltqz krb5kdc[9366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:30.601596 9379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:30.601994 9379 webserver.cc:460] Webserver started at http://127.0.0.1:33205/ using document root <none> and password file <none>
I20260927 07:08:30.602167 9379 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:30.602205 9379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:30.602293 9379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:30.603308 9379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "d8ede0af2f6442f99fb667d91d793409"
format_stamp: "Formatted at 2026-09-27 07:08:30 on dist-test-slave-ltqz"
server_key: "6d3d3d434473c9cae38cd946128eacf0"
server_key_iv: "968210b78e41fcc618ceb35f35d7c15f"
server_key_version: "encryptionkey@0"
I20260927 07:08:30.603708 9379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "d8ede0af2f6442f99fb667d91d793409"
format_stamp: "Formatted at 2026-09-27 07:08:30 on dist-test-slave-ltqz"
server_key: "6d3d3d434473c9cae38cd946128eacf0"
server_key_iv: "968210b78e41fcc618ceb35f35d7c15f"
server_key_version: "encryptionkey@0"
I20260927 07:08:30.605007 9379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:30.605684 9394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:30.605836 9379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:30.605922 9379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "d8ede0af2f6442f99fb667d91d793409"
format_stamp: "Formatted at 2026-09-27 07:08:30 on dist-test-slave-ltqz"
server_key: "6d3d3d434473c9cae38cd946128eacf0"
server_key_iv: "968210b78e41fcc618ceb35f35d7c15f"
server_key_version: "encryptionkey@0"
I20260927 07:08:30.605976 9379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:30.614689 9379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:30.616477 9379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:30.616582 9379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:30.620265 9379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35525
I20260927 07:08:30.620338 9446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35525 every 8 connection(s)
I20260927 07:08:30.620652 9379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:30.621476 9447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:30.623749 9447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409: Bootstrap starting.
I20260927 07:08:30.624313 9447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:30.624630 9447 log.cc:824] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:30.625311 9447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409: No bootstrap required, opened a new log
I20260927 07:08:30.626855 9447 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260927 07:08:30.627010 9447 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:30.627055 9447 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ede0af2f6442f99fb667d91d793409, State: Initialized, Role: FOLLOWER
I20260927 07:08:30.627216 9447 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260927 07:08:30.627297 9447 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:30.627336 9447 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:30.627411 9447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:30.627924 9447 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260927 07:08:30.628041 9447 leader_election.cc:302] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ede0af2f6442f99fb667d91d793409; no voters:
I20260927 07:08:30.628207 9447 leader_election.cc:288] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:30.628242 9450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:30.628397 9450 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [term 1 LEADER]: Becoming Leader. State: Replica: d8ede0af2f6442f99fb667d91d793409, State: Running, Role: LEADER
I20260927 07:08:30.628460 9447 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:30.628495 9450 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260927 07:08:30.628721 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9379
I20260927 07:08:30.628870 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:30.628885 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ede0af2f6442f99fb667d91d793409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260927 07:08:30.629040 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:30.629053 9452 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ede0af2f6442f99fb667d91d793409. Latest consensus state: current_term: 1 leader_uuid: "d8ede0af2f6442f99fb667d91d793409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ede0af2f6442f99fb667d91d793409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260927 07:08:30.629086 21513 external_mini_cluster.cc:1428] Setting key 471717696e59e3e0c9a6f36c38a486da
I20260927 07:08:30.629112 9452 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:30.629449 9454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:30.629961 9454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:30.631528 9454 catalog_manager.cc:1347] Generated new cluster ID: 3a32a622e5ad45cb8afb2f3027cee73b
I20260927 07:08:30.631579 9454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:30.636211 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:30.629744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35845 (local address 127.0.0.1:35525)
0927 07:08:30.630606 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:30.630615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:30.630645 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:08:30.630756 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:30.630762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:30.630817 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:30.630964 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:30.631220 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:30.632615 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:30.633782 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:30.634407 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:30.634849 (+ 442us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:30.635300 (+ 451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:30.635304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:30.635304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:30.635539 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:30.635578 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:30.635596 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:30.635899 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:30.635942 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 07:08:30.635972 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":80,"threads_started":1}
I20260927 07:08:30.637132 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:30.641767 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:30.638527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44245 (local address 127.0.0.1:35525)
0927 07:08:30.638564 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:30.638566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:30.638702 (+ 136us) server_negotiation.cc:407] Connection header received
0927 07:08:30.638776 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:30.638777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:30.638789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:30.638863 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:30.639003 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:30.639823 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:30.640532 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:30.640741 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:30.640818 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:30.641488 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:30.641489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:30.641490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:30.641494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:30.641510 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:30.641514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:30.641588 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:30.641654 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:08:30.641667 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:08:30.647606 9454 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:30.648345 9454 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:30.653759 9454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ede0af2f6442f99fb667d91d793409: Generated new TSK 0
I20260927 07:08:30.653960 9454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:30.657740 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9379
W20260927 07:08:30.661623 9364 connection.cc:535] client connection to 127.0.0.1:35525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35525): Connection reset by peer (error 104)
2026-09-27T07:08:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:30 dist-test-slave-ltqz krb5kdc[9478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:30 dist-test-slave-ltqz krb5kdc[9478](info): set up 2 sockets
Sep 27 07:08:30 dist-test-slave-ltqz krb5kdc[9478](info): commencing operation
krb5kdc: starting...
W20260927 07:08:31.691536 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:31 dist-test-slave-ltqz krb5kdc[9478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:31.810575 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:31.892522 9491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:31.892827 9491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:31.892875 9491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:31.894220 9491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:31.894316 9491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:31.894345 9491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:31.894375 9491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:31.894411 9491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:31.896126 9491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:31.896451 9491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:31.896706 9491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:31.902119 9499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:31.902160 9496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:31.902127 9497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:31.902200 9491 server_base.cc:1034] running on GCE node
I20260927 07:08:31.902570 9491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:31.902890 9491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:31.904059 9491 hybrid_clock.cc:648] HybridClock initialized: now 1790492911904057 us; error 37 us; skew 500 ppm
Sep 27 07:08:31 dist-test-slave-ltqz krb5kdc[9478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:31.906606 9491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:31.907117 9491 webserver.cc:460] Webserver started at http://127.0.0.1:39567/ using document root <none> and password file <none>
I20260927 07:08:31.907346 9491 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:31.907476 9491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:31.907636 9491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:31.908746 9491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "a9732080a951479c80a0fac37f479e9c"
format_stamp: "Formatted at 2026-09-27 07:08:31 on dist-test-slave-ltqz"
server_key: "3a6fba62cddfbd8bd2a484c77421ccab"
server_key_iv: "69cb1c7d85cc44e2fc7e4b93f558b906"
server_key_version: "encryptionkey@0"
I20260927 07:08:31.909121 9491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "a9732080a951479c80a0fac37f479e9c"
format_stamp: "Formatted at 2026-09-27 07:08:31 on dist-test-slave-ltqz"
server_key: "3a6fba62cddfbd8bd2a484c77421ccab"
server_key_iv: "69cb1c7d85cc44e2fc7e4b93f558b906"
server_key_version: "encryptionkey@0"
I20260927 07:08:31.910487 9491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:31.911325 9506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:31.911556 9491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:31.911665 9491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "a9732080a951479c80a0fac37f479e9c"
format_stamp: "Formatted at 2026-09-27 07:08:31 on dist-test-slave-ltqz"
server_key: "3a6fba62cddfbd8bd2a484c77421ccab"
server_key_iv: "69cb1c7d85cc44e2fc7e4b93f558b906"
server_key_version: "encryptionkey@0"
I20260927 07:08:31.911734 9491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:31.915096 9491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:31.917033 9491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:31.917150 9491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:31.921156 9491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34693
I20260927 07:08:31.921227 9558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34693 every 8 connection(s)
I20260927 07:08:31.921561 9491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:31.922482 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:31.924795 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c: Bootstrap starting.
I20260927 07:08:31.925216 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9491
I20260927 07:08:31.925355 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:31.925447 9559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:31.925586 21513 external_mini_cluster.cc:1428] Setting key 10459048e7f597a1f88eaeed5e0be681
I20260927 07:08:31.925799 9559 log.cc:824] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:31.926636 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c: No bootstrap required, opened a new log
I20260927 07:08:31.928395 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:31.926190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44809 (local address 127.0.0.1:34693)
0927 07:08:31.926369 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:31.926378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:31.926416 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:08:31.927028 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:31.927037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:31.927071 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:31.927224 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:31.927964 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:31.927968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:31.927968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:31.927973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:31.928016 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:31.928031 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:31.928122 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:31.928158 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 07:08:31.928198 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260927 07:08:31.929199 9559 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260927 07:08:31.929359 9559 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:31.929405 9559 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9732080a951479c80a0fac37f479e9c, State: Initialized, Role: FOLLOWER
I20260927 07:08:31.929546 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:31.929548 9559 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260927 07:08:31.929637 9559 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:31.929682 9559 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:31.929730 9559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:31.930279 9559 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260927 07:08:31.930392 9559 leader_election.cc:302] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9732080a951479c80a0fac37f479e9c; no voters:
I20260927 07:08:31.930590 9559 leader_election.cc:288] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:31.930612 9565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:31.930723 9565 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [term 1 LEADER]: Becoming Leader. State: Replica: a9732080a951479c80a0fac37f479e9c, State: Running, Role: LEADER
I20260927 07:08:31.930842 9559 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:31.930866 9565 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260927 07:08:31.931332 9568 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9732080a951479c80a0fac37f479e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } } }
I20260927 07:08:31.931466 9568 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:31.931658 9569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9732080a951479c80a0fac37f479e9c. Latest consensus state: current_term: 1 leader_uuid: "a9732080a951479c80a0fac37f479e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9732080a951479c80a0fac37f479e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } } }
I20260927 07:08:31.931779 9569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:31.932425 9577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:31.932680 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:31.931038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42875 (local address 127.0.0.1:34693)
0927 07:08:31.931078 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:31.931080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:31.931409 (+ 329us) server_negotiation.cc:407] Connection header received
0927 07:08:31.931635 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:31.931636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:31.931640 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:31.931675 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:31.932376 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:31.932377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:31.932378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:31.932382 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:31.932395 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:31.932399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:31.932461 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:31.932609 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 07:08:31.932622 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:08:31.932996 9577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:31.934593 9577 catalog_manager.cc:1347] Generated new cluster ID: 2775382a067848aca92ae87c56f10b3f
I20260927 07:08:31.934660 9577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:31.945502 9577 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:31.945712 9577 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:31.953730 9577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9732080a951479c80a0fac37f479e9c: Generated new TSK 0
I20260927 07:08:31.953944 9577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:31.959043 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9491
2026-09-27T07:08:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:32 dist-test-slave-ltqz krb5kdc[9590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:32 dist-test-slave-ltqz krb5kdc[9590](info): set up 2 sockets
Sep 27 07:08:32 dist-test-slave-ltqz krb5kdc[9590](info): commencing operation
krb5kdc: starting...
W20260927 07:08:32.999512 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:33.117506 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:33.197002 9603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:33.197263 9603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:33.197301 9603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:33.198472 9603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:33.198552 9603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:33.198576 9603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:33.198598 9603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:33.198628 9603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:33.200183 9603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:33.200481 9603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:33.200709 9603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:33.205765 9603 server_base.cc:1034] running on GCE node
W20260927 07:08:33.205749 9609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:33.205734 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:33.205719 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:33.206216 9603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:33.206490 9603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:33.207648 9603 hybrid_clock.cc:648] HybridClock initialized: now 1790492913207633 us; error 33 us; skew 500 ppm
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:33.210835 9603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:33.211416 9603 webserver.cc:460] Webserver started at http://127.0.0.1:43131/ using document root <none> and password file <none>
I20260927 07:08:33.211689 9603 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:33.211757 9603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:33.211905 9603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:33.213395 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "c5774f7da3b34242aa8d750fb6125853"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-ltqz"
server_key: "33eafebf541d476cb57f601b550e55bd"
server_key_iv: "fc77de154642fa1777aab5354513010e"
server_key_version: "encryptionkey@0"
I20260927 07:08:33.213825 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "c5774f7da3b34242aa8d750fb6125853"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-ltqz"
server_key: "33eafebf541d476cb57f601b550e55bd"
server_key_iv: "fc77de154642fa1777aab5354513010e"
server_key_version: "encryptionkey@0"
I20260927 07:08:33.215621 9603 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 07:08:33.217512 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:33.217725 9603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:08:33.217823 9603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "c5774f7da3b34242aa8d750fb6125853"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-ltqz"
server_key: "33eafebf541d476cb57f601b550e55bd"
server_key_iv: "fc77de154642fa1777aab5354513010e"
server_key_version: "encryptionkey@0"
I20260927 07:08:33.217902 9603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:33.241906 9603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:33.243834 9603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:33.243947 9603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:33.247912 9603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44663
I20260927 07:08:33.247982 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44663 every 8 connection(s)
I20260927 07:08:33.248358 9603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:33.249239 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:33.251622 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853: Bootstrap starting.
I20260927 07:08:33.252091 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9603
I20260927 07:08:33.252225 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:33.252406 9671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:33.252427 21513 external_mini_cluster.cc:1428] Setting key 19c0d4957e376d469f554a317f247f97
I20260927 07:08:33.252745 9671 log.cc:824] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:33.254292 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853: No bootstrap required, opened a new log
I20260927 07:08:33.256588 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260927 07:08:33.256795 9671 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:33.256830 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5774f7da3b34242aa8d750fb6125853, State: Initialized, Role: FOLLOWER
I20260927 07:08:33.256950 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260927 07:08:33.257021 9671 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:33.257050 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:33.257095 9671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:33.257800 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260927 07:08:33.257927 9671 leader_election.cc:302] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5774f7da3b34242aa8d750fb6125853; no voters:
I20260927 07:08:33.258145 9671 leader_election.cc:288] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:33.258172 9676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:33.258301 9676 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [term 1 LEADER]: Becoming Leader. State: Replica: c5774f7da3b34242aa8d750fb6125853, State: Running, Role: LEADER
I20260927 07:08:33.258390 9676 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } }
I20260927 07:08:33.258420 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:33.258872 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5774f7da3b34242aa8d750fb6125853. Latest consensus state: current_term: 1 leader_uuid: "c5774f7da3b34242aa8d750fb6125853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } } }
I20260927 07:08:33.258950 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:33.259209 9680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:33.259727 9680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:08:33.260565 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5774f7da3b34242aa8d750fb6125853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5774f7da3b34242aa8d750fb6125853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44663 } } }
I20260927 07:08:33.260860 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:33.261724 9680 catalog_manager.cc:1347] Generated new cluster ID: 7652375c182449e2b7dae4cb74c21a0c
I20260927 07:08:33.261780 9680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:33.264447 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:33.252893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55021 (local address 127.0.0.1:44663)
0927 07:08:33.253075 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:33.253082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:33.253106 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:08:33.253536 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:33.253542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:33.253587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:33.253711 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:33.254335 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.255709 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.256690 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.257313 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.257426 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:33.260517 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:33.260522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:33.260523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:33.260539 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:33.262589 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.263053 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.263056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.263057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.263171 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.263465 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.263467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.263467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.263768 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:33.263919 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:33.264056 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 07:08:33.264189 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":59,"threads_started":1}
I20260927 07:08:33.265658 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:33.273916 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:33.267045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46319 (local address 127.0.0.1:44663)
0927 07:08:33.267080 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:33.267082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:33.267179 (+ 97us) server_negotiation.cc:407] Connection header received
0927 07:08:33.267272 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:33.267273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:33.267282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:33.267307 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:33.267969 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.268703 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.269657 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.269948 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.269983 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:33.271393 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:33.271394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:33.271394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:33.271400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:33.272848 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.273189 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.273190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.273191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.273244 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.273412 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.273412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.273413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.273519 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:33.273578 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:33.273755 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 07:08:33.273824 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:08:33.292156 9680 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:33.292912 9680 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:33.296471 9680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5774f7da3b34242aa8d750fb6125853: Generated new TSK 0
I20260927 07:08:33.296651 9680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:33.303622 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9603
2026-09-27T07:08:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9702](info): set up 2 sockets
Sep 27 07:08:33 dist-test-slave-ltqz krb5kdc[9702](info): commencing operation
krb5kdc: starting...
W20260927 07:08:34.339609 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:34 dist-test-slave-ltqz krb5kdc[9702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:34.458598 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:34.540887 9715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:34.541103 9715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:34.541128 9715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:34.542553 9715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:34.542637 9715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:34.542655 9715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:34.542670 9715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:34.542685 9715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:34.544346 9715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:34.544593 9715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:34.544929 9715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:34.550052 9723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:34.550086 9721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:34.550130 9720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:34.550195 9715 server_base.cc:1034] running on GCE node
I20260927 07:08:34.550639 9715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:34.550972 9715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:34.552134 9715 hybrid_clock.cc:648] HybridClock initialized: now 1790492914552134 us; error 40 us; skew 500 ppm
Sep 27 07:08:34 dist-test-slave-ltqz krb5kdc[9702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:34.554644 9715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:34.555099 9715 webserver.cc:460] Webserver started at http://127.0.0.1:46669/ using document root <none> and password file <none>
I20260927 07:08:34.555297 9715 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:34.555346 9715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:34.555509 9715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:34.556523 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "6cd389ce336740c9bfae1ee339f688df"
format_stamp: "Formatted at 2026-09-27 07:08:34 on dist-test-slave-ltqz"
server_key: "1726fb1b55edcea1f402932735b5449c"
server_key_iv: "282b8ac63d6c382fa38d8245ec4c59da"
server_key_version: "encryptionkey@0"
I20260927 07:08:34.556813 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "6cd389ce336740c9bfae1ee339f688df"
format_stamp: "Formatted at 2026-09-27 07:08:34 on dist-test-slave-ltqz"
server_key: "1726fb1b55edcea1f402932735b5449c"
server_key_iv: "282b8ac63d6c382fa38d8245ec4c59da"
server_key_version: "encryptionkey@0"
I20260927 07:08:34.558025 9715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:34.558753 9730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:34.558928 9715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:34.559019 9715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "6cd389ce336740c9bfae1ee339f688df"
format_stamp: "Formatted at 2026-09-27 07:08:34 on dist-test-slave-ltqz"
server_key: "1726fb1b55edcea1f402932735b5449c"
server_key_iv: "282b8ac63d6c382fa38d8245ec4c59da"
server_key_version: "encryptionkey@0"
I20260927 07:08:34.559077 9715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:34.577129 9715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:34.579465 9715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:34.579592 9715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:34.583334 9715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46679
I20260927 07:08:34.583446 9782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46679 every 8 connection(s)
I20260927 07:08:34.583828 9715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:34.584707 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:34.587030 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df: Bootstrap starting.
I20260927 07:08:34.587572 9783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:34.587816 9783 log.cc:824] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:34.588497 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df: No bootstrap required, opened a new log
I20260927 07:08:34.590044 9783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260927 07:08:34.590185 9783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:34.590207 9783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd389ce336740c9bfae1ee339f688df, State: Initialized, Role: FOLLOWER
I20260927 07:08:34.590364 9783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260927 07:08:34.590456 9783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:34.590479 9783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:34.590508 9783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:34.591008 9783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260927 07:08:34.591091 9783 leader_election.cc:302] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cd389ce336740c9bfae1ee339f688df; no voters:
I20260927 07:08:34.591223 9783 leader_election.cc:288] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:34.591295 9786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:34.591508 9786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [term 1 LEADER]: Becoming Leader. State: Replica: 6cd389ce336740c9bfae1ee339f688df, State: Running, Role: LEADER
I20260927 07:08:34.591537 9783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:34.591598 9786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260927 07:08:34.591991 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cd389ce336740c9bfae1ee339f688df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260927 07:08:34.592078 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:34.592206 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd389ce336740c9bfae1ee339f688df. Latest consensus state: current_term: 1 leader_uuid: "6cd389ce336740c9bfae1ee339f688df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd389ce336740c9bfae1ee339f688df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260927 07:08:34.592300 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:34.592309 9794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:34.592882 9794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:34.593748 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9715
I20260927 07:08:34.593884 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:34.594085 21513 external_mini_cluster.cc:1428] Setting key 3d0cd1317fc7e48bde28b90d1f9f6eb6
I20260927 07:08:34.594722 9794 catalog_manager.cc:1347] Generated new cluster ID: 95e89389aa194170aee3f9f704a6cd0d
I20260927 07:08:34.594784 9794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:34.600342 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:34.594552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59183 (local address 127.0.0.1:46679)
0927 07:08:34.594722 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:34.594732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:34.594905 (+ 173us) server_negotiation.cc:407] Connection header received
0927 07:08:34.595029 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:34.595036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:34.595087 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:34.595207 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:34.595524 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:34.596890 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:34.597957 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:34.598554 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:34.598646 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:34.599521 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:34.599525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:34.599525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:34.599723 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:34.599759 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:34.599773 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:34.599824 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:34.600064 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 07:08:34.600122 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260927 07:08:34.601068 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:34.603108 9794 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:34.603796 9794 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:34.606343 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:34.602699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39373 (local address 127.0.0.1:46679)
0927 07:08:34.602733 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:34.602734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:34.602862 (+ 128us) server_negotiation.cc:407] Connection header received
0927 07:08:34.602929 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:34.602930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:34.602940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:34.603013 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:34.603164 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:34.603881 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:34.604948 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:34.605250 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:34.605296 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:34.606063 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:34.606064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:34.606065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:34.606071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:34.606086 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:34.606091 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:34.606143 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:34.606221 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 07:08:34.606262 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:34.612752 9794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cd389ce336740c9bfae1ee339f688df: Generated new TSK 0
I20260927 07:08:34.612972 9794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:34.616720 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9715
2026-09-27T07:08:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:34 dist-test-slave-ltqz krb5kdc[9814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:34 dist-test-slave-ltqz krb5kdc[9814](info): set up 2 sockets
Sep 27 07:08:34 dist-test-slave-ltqz krb5kdc[9814](info): commencing operation
krb5kdc: starting...
W20260927 07:08:35.643724 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:35 dist-test-slave-ltqz krb5kdc[9814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:35.758415 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:35.838703 9827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:35.839030 9827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:35.839097 9827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:35.840569 9827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:35.840637 9827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:35.840663 9827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:35.840698 9827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:35.840723 9827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:35.842159 9827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:35.842427 9827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:35.842650 9827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:35.847826 9833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:35.847790 9835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:35.847790 9832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:35.848089 9827 server_base.cc:1034] running on GCE node
I20260927 07:08:35.848369 9827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:35.848704 9827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:35.849893 9827 hybrid_clock.cc:648] HybridClock initialized: now 1790492915849882 us; error 33 us; skew 500 ppm
Sep 27 07:08:35 dist-test-slave-ltqz krb5kdc[9814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:35.857569 9827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:35.858057 9827 webserver.cc:460] Webserver started at http://127.0.0.1:33651/ using document root <none> and password file <none>
I20260927 07:08:35.858294 9827 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:35.858366 9827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:35.858521 9827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:35.859720 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "dd531f6b92c941b5881ce5abd6f5b759"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-ltqz"
server_key: "ddbff360e33fec765ee36e5c5411b870"
server_key_iv: "06b1a7ab88554d91f35d2eab12721207"
server_key_version: "encryptionkey@0"
I20260927 07:08:35.860085 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "dd531f6b92c941b5881ce5abd6f5b759"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-ltqz"
server_key: "ddbff360e33fec765ee36e5c5411b870"
server_key_iv: "06b1a7ab88554d91f35d2eab12721207"
server_key_version: "encryptionkey@0"
I20260927 07:08:35.861444 9827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:35.862277 9842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:35.862457 9827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:35.862561 9827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "dd531f6b92c941b5881ce5abd6f5b759"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-ltqz"
server_key: "ddbff360e33fec765ee36e5c5411b870"
server_key_iv: "06b1a7ab88554d91f35d2eab12721207"
server_key_version: "encryptionkey@0"
I20260927 07:08:35.862655 9827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:35.866014 9827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:35.868110 9827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:35.868266 9827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:35.872112 9827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41783
I20260927 07:08:35.872141 9894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41783 every 8 connection(s)
I20260927 07:08:35.872465 9827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:35.873108 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9827
I20260927 07:08:35.873237 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:35.873438 21513 external_mini_cluster.cc:1428] Setting key f795d94ac915c65c74c944767e3b925a
I20260927 07:08:35.873488 9895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:35.875933 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:35.873977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34573 (local address 127.0.0.1:41783)
0927 07:08:35.874153 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:35.874160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:35.874185 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:08:35.874618 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:35.874624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:35.874648 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:35.874776 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:35.875440 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:35.875443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:35.875443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:35.875447 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:35.875482 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:35.875492 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:35.875547 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:35.875665 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 07:08:35.875724 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260927 07:08:35.877012 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:35.877019 9895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759: Bootstrap starting.
I20260927 07:08:35.877615 9895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:35.877919 9895 log.cc:824] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:35.878866 9895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759: No bootstrap required, opened a new log
I20260927 07:08:35.879526 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:35.878422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44735 (local address 127.0.0.1:41783)
0927 07:08:35.878459 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:35.878461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:35.878617 (+ 156us) server_negotiation.cc:407] Connection header received
0927 07:08:35.878660 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:35.878661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:35.878666 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:35.878718 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:35.879289 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:35.879290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:35.879291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:35.879295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:35.879310 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:35.879314 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:35.879395 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:35.879456 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 07:08:35.879468 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:35.881027 9895 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 07:08:35.881163 9895 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:35.881186 9895 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd531f6b92c941b5881ce5abd6f5b759, State: Initialized, Role: FOLLOWER
I20260927 07:08:35.881313 9895 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 07:08:35.881404 9895 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:35.881428 9895 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:35.881455 9895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:35.881949 9895 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 07:08:35.882035 9895 leader_election.cc:302] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd531f6b92c941b5881ce5abd6f5b759; no voters:
I20260927 07:08:35.882161 9895 leader_election.cc:288] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:35.882246 9905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:35.882424 9905 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [term 1 LEADER]: Becoming Leader. State: Replica: dd531f6b92c941b5881ce5abd6f5b759, State: Running, Role: LEADER
I20260927 07:08:35.882457 9895 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:35.882525 9905 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 07:08:35.882879 9906 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd531f6b92c941b5881ce5abd6f5b759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } } }
I20260927 07:08:35.882987 9906 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:35.882884 9907 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd531f6b92c941b5881ce5abd6f5b759. Latest consensus state: current_term: 1 leader_uuid: "dd531f6b92c941b5881ce5abd6f5b759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd531f6b92c941b5881ce5abd6f5b759" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } } }
I20260927 07:08:35.883070 9907 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:35.883268 9913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:35.883788 9913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:35.885377 9913 catalog_manager.cc:1347] Generated new cluster ID: 81d6f1f9dbd9472d9730b55064cff7f3
I20260927 07:08:35.885452 9913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:35.897504 9913 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:35.897723 9913 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:35.906080 9913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd531f6b92c941b5881ce5abd6f5b759: Generated new TSK 0
I20260927 07:08:35.906281 9913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:35.915084 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9827
2026-09-27T07:08:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:35 dist-test-slave-ltqz krb5kdc[9926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:35 dist-test-slave-ltqz krb5kdc[9926](info): set up 2 sockets
Sep 27 07:08:35 dist-test-slave-ltqz krb5kdc[9926](info): commencing operation
krb5kdc: starting...
W20260927 07:08:36.947621 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:37.064551 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:37.146245 9939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:37.146479 9939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:37.146503 9939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:37.147771 9939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:37.147817 9939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:37.147831 9939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:37.147876 9939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:37.147907 9939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:37.149410 9939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:37.149685 9939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:37.149919 9939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:37.154740 9945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:37.154719 9947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:37.154712 9944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:37.154739 9939 server_base.cc:1034] running on GCE node
I20260927 07:08:37.155218 9939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:37.155545 9939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:37.156697 9939 hybrid_clock.cc:648] HybridClock initialized: now 1790492917156684 us; error 31 us; skew 500 ppm
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:37.159282 9939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:37.159744 9939 webserver.cc:460] Webserver started at http://127.0.0.1:45533/ using document root <none> and password file <none>
I20260927 07:08:37.159942 9939 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:37.159981 9939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:37.160070 9939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:37.161049 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "be8b99b4d8944892a16a66dc93c3abeb"
format_stamp: "Formatted at 2026-09-27 07:08:37 on dist-test-slave-ltqz"
server_key: "f626b77dd26fc93b316be43d52c9c2ea"
server_key_iv: "bc3e7e5d2a5c4823c92cca075e57e7b1"
server_key_version: "encryptionkey@0"
I20260927 07:08:37.161357 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "be8b99b4d8944892a16a66dc93c3abeb"
format_stamp: "Formatted at 2026-09-27 07:08:37 on dist-test-slave-ltqz"
server_key: "f626b77dd26fc93b316be43d52c9c2ea"
server_key_iv: "bc3e7e5d2a5c4823c92cca075e57e7b1"
server_key_version: "encryptionkey@0"
I20260927 07:08:37.162577 9939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:37.163249 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:37.163446 9939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:37.163546 9939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "be8b99b4d8944892a16a66dc93c3abeb"
format_stamp: "Formatted at 2026-09-27 07:08:37 on dist-test-slave-ltqz"
server_key: "f626b77dd26fc93b316be43d52c9c2ea"
server_key_iv: "bc3e7e5d2a5c4823c92cca075e57e7b1"
server_key_version: "encryptionkey@0"
I20260927 07:08:37.163630 9939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:37.177131 9939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:37.179538 9939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:37.179700 9939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:37.183611 9939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43245
I20260927 07:08:37.183655 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43245 every 8 connection(s)
I20260927 07:08:37.183988 9939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:37.184819 10007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:37.187129 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb: Bootstrap starting.
I20260927 07:08:37.187783 10007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:37.188086 10007 log.cc:824] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:37.188844 10007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb: No bootstrap required, opened a new log
I20260927 07:08:37.189620 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 9939
I20260927 07:08:37.189764 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:37.189957 21513 external_mini_cluster.cc:1428] Setting key dc0c9d57f845e3111b41ce1778e3e8c0
I20260927 07:08:37.190784 10007 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260927 07:08:37.190943 10007 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:37.191004 10007 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8b99b4d8944892a16a66dc93c3abeb, State: Initialized, Role: FOLLOWER
I20260927 07:08:37.191162 10007 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260927 07:08:37.191262 10007 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:37.191334 10007 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:37.191458 10007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:37.192047 10007 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260927 07:08:37.192178 10007 leader_election.cc:302] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8b99b4d8944892a16a66dc93c3abeb; no voters:
I20260927 07:08:37.192350 10007 leader_election.cc:288] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:37.192386 10012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:37.192540 10012 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [term 1 LEADER]: Becoming Leader. State: Replica: be8b99b4d8944892a16a66dc93c3abeb, State: Running, Role: LEADER
I20260927 07:08:37.192622 10007 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:37.192718 10012 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } }
I20260927 07:08:37.193130 10014 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8b99b4d8944892a16a66dc93c3abeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } } }
I20260927 07:08:37.193549 10014 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:37.193711 10014 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8b99b4d8944892a16a66dc93c3abeb. Latest consensus state: current_term: 1 leader_uuid: "be8b99b4d8944892a16a66dc93c3abeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b99b4d8944892a16a66dc93c3abeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43245 } } }
I20260927 07:08:37.193771 10014 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:37.193995 10023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:37.194990 10023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:37.196887 10023 catalog_manager.cc:1347] Generated new cluster ID: 7096bbf7dc5f49c7b68aafea005a568c
I20260927 07:08:37.196938 10023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:08:37.202011 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.190474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:44317 (local address 127.0.0.1:43245)
0927 07:08:37.190694 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.190704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.190741 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:08:37.190834 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.190839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.190884 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.191023 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.191832 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.193134 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.194312 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.194880 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.195005 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.197880 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.197885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.197885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.197898 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.200033 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.200490 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.200494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.200494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.200631 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.200873 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.200874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.200874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.201214 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.201393 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.201507 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.201622 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260927 07:08:37.203532 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:37.215545 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.207740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:38559 (local address 127.0.0.1:43245)
0927 07:08:37.207800 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.207803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.207989 (+ 186us) server_negotiation.cc:407] Connection header received
0927 07:08:37.208039 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.208040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.208052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.208108 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.208730 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.209514 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.210582 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.210909 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.210950 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.212505 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.212507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.212507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.212517 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.214065 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.214501 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.214502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.214503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.214569 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.214803 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.214804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.214804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.214931 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.215006 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.215243 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.215307 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:08:37.236471 10023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:37.237025 10023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:37.238801 10023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8b99b4d8944892a16a66dc93c3abeb: Generated new TSK 0
I20260927 07:08:37.238989 10023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:37.243086 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 9939
W20260927 07:08:37.247251 9924 connection.cc:535] client connection to 127.0.0.1:43245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43245): Connection reset by peer (error 104)
2026-09-27T07:08:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[10038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[10038](info): set up 2 sockets
Sep 27 07:08:37 dist-test-slave-ltqz krb5kdc[10038](info): commencing operation
krb5kdc: starting...
W20260927 07:08:38.275528 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:38 dist-test-slave-ltqz krb5kdc[10038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:38.394186 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:38.477296 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:38.477514 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:38.477538 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:38.478721 10051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:38.478767 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:38.478781 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:38.478796 10051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:38.478850 10051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:38.480414 10051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:38.480679 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:38.480913 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:38.485698 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:38.485734 10059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:38.485834 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:38.486095 10051 server_base.cc:1034] running on GCE node
I20260927 07:08:38.486310 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:38.486541 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:38.487720 10051 hybrid_clock.cc:648] HybridClock initialized: now 1790492918487705 us; error 29 us; skew 500 ppm
Sep 27 07:08:38 dist-test-slave-ltqz krb5kdc[10038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:38.490043 10051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:38.490418 10051 webserver.cc:460] Webserver started at http://127.0.0.1:36393/ using document root <none> and password file <none>
I20260927 07:08:38.490592 10051 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:38.490630 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:38.490720 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:38.491791 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "fa1b53c43211443b906b83b61750a197"
format_stamp: "Formatted at 2026-09-27 07:08:38 on dist-test-slave-ltqz"
server_key: "c135d88aeba5e3ba1f101978251f1527"
server_key_iv: "aeabf58b5d8bafe3e6dc810fa71dcdf4"
server_key_version: "encryptionkey@0"
I20260927 07:08:38.492086 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "fa1b53c43211443b906b83b61750a197"
format_stamp: "Formatted at 2026-09-27 07:08:38 on dist-test-slave-ltqz"
server_key: "c135d88aeba5e3ba1f101978251f1527"
server_key_iv: "aeabf58b5d8bafe3e6dc810fa71dcdf4"
server_key_version: "encryptionkey@0"
I20260927 07:08:38.493310 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:38.493989 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:38.494161 10051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:38.494272 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "fa1b53c43211443b906b83b61750a197"
format_stamp: "Formatted at 2026-09-27 07:08:38 on dist-test-slave-ltqz"
server_key: "c135d88aeba5e3ba1f101978251f1527"
server_key_iv: "aeabf58b5d8bafe3e6dc810fa71dcdf4"
server_key_version: "encryptionkey@0"
I20260927 07:08:38.494347 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:38.510979 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:38.512938 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:38.513103 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:38.517148 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46313
I20260927 07:08:38.517185 10118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46313 every 8 connection(s)
I20260927 07:08:38.517537 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:38.518410 10119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:38.518872 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10051
I20260927 07:08:38.519006 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:38.519207 21513 external_mini_cluster.cc:1428] Setting key eb1ff2a0c18fc990353a33520f353f0d
I20260927 07:08:38.521327 10119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: Bootstrap starting.
I20260927 07:08:38.521999 10119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:38.522332 10119 log.cc:824] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:38.523044 10119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: No bootstrap required, opened a new log
I20260927 07:08:38.524713 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:38.519782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:49343 (local address 127.0.0.1:46313)
0927 07:08:38.519954 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:38.519961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:38.519987 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:08:38.520383 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:38.520389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:38.520435 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:38.520550 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:38.520804 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:38.521716 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:38.522808 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:38.523249 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:38.523333 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:38.524045 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:38.524050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:38.524050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:38.524231 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:38.524260 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:38.524271 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:38.524336 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:38.524459 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 07:08:38.524502 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260927 07:08:38.525405 10119 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260927 07:08:38.525563 10119 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:38.525600 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:38.525612 10119 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1b53c43211443b906b83b61750a197, State: Initialized, Role: FOLLOWER
I20260927 07:08:38.525744 10119 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260927 07:08:38.525836 10119 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:38.525876 10119 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:38.525945 10119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:38.526502 10119 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260927 07:08:38.526638 10119 leader_election.cc:302] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa1b53c43211443b906b83b61750a197; no voters:
I20260927 07:08:38.526818 10119 leader_election.cc:288] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:38.526870 10128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:38.527040 10128 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [term 1 LEADER]: Becoming Leader. State: Replica: fa1b53c43211443b906b83b61750a197, State: Running, Role: LEADER
I20260927 07:08:38.527074 10119 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:38.527137 10128 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260927 07:08:38.527443 10128 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa1b53c43211443b906b83b61750a197. Latest consensus state: current_term: 1 leader_uuid: "fa1b53c43211443b906b83b61750a197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260927 07:08:38.527454 10131 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa1b53c43211443b906b83b61750a197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1b53c43211443b906b83b61750a197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260927 07:08:38.527591 10128 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:38.527967 10131 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197 [sys.catalog]: This master's current role is: LEADER
W20260927 07:08:38.528901 10145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:38.529003 10145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:38.529093 10138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:38.529237 10138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:38.530751 10138 catalog_manager.cc:1347] Generated new cluster ID: 6b24390dc278402e870d3c983e1481c0
I20260927 07:08:38.530817 10138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:38.531486 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:38.526949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:32965 (local address 127.0.0.1:46313)
0927 07:08:38.526988 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:38.526991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:38.527577 (+ 586us) server_negotiation.cc:407] Connection header received
0927 07:08:38.527691 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:38.527692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:38.527704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:38.527766 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:38.527928 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:38.528680 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:38.529425 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:38.529644 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:38.529734 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:38.530483 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:38.530484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:38.530485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:38.530489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:38.530505 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:38.530510 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:38.530542 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:38.530692 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 07:08:38.530705 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:08:38.537055 10138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:38.537812 10138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:38.545514 10138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa1b53c43211443b906b83b61750a197: Generated new TSK 0
I20260927 07:08:38.545874 10138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:38.548969 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10051
W20260927 07:08:38.553609 10036 connection.cc:535] client connection to 127.0.0.1:46313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46313): Connection reset by peer (error 104)
2026-09-27T07:08:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:38 dist-test-slave-ltqz krb5kdc[10150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:38 dist-test-slave-ltqz krb5kdc[10150](info): set up 2 sockets
Sep 27 07:08:38 dist-test-slave-ltqz krb5kdc[10150](info): commencing operation
krb5kdc: starting...
W20260927 07:08:39.591452 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:39 dist-test-slave-ltqz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:39.711079 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:39.791013 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:39.791287 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:39.791326 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:39.792511 10163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:39.792610 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:39.792637 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:39.792665 10163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:39.792695 10163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:39.794206 10163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:39.794492 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:39.794732 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:39.799499 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:39.799469 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:39.799542 10163 server_base.cc:1034] running on GCE node
W20260927 07:08:39.799451 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:39.799921 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:39.800221 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:39.801368 10163 hybrid_clock.cc:648] HybridClock initialized: now 1790492919801351 us; error 39 us; skew 500 ppm
Sep 27 07:08:39 dist-test-slave-ltqz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:39.803926 10163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:39.804402 10163 webserver.cc:460] Webserver started at http://127.0.0.1:36785/ using document root <none> and password file <none>
I20260927 07:08:39.804593 10163 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:39.804634 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:39.804723 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:39.805729 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "f969192be2a2471198c1975e81f5aa0b"
format_stamp: "Formatted at 2026-09-27 07:08:39 on dist-test-slave-ltqz"
server_key: "788313845655fa83e613f0ae40d22654"
server_key_iv: "d5db1c6ed33567df6692971aa1c33bcf"
server_key_version: "encryptionkey@0"
I20260927 07:08:39.806027 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "f969192be2a2471198c1975e81f5aa0b"
format_stamp: "Formatted at 2026-09-27 07:08:39 on dist-test-slave-ltqz"
server_key: "788313845655fa83e613f0ae40d22654"
server_key_iv: "d5db1c6ed33567df6692971aa1c33bcf"
server_key_version: "encryptionkey@0"
I20260927 07:08:39.807312 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:39.808113 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:39.808293 10163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:39.808409 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "f969192be2a2471198c1975e81f5aa0b"
format_stamp: "Formatted at 2026-09-27 07:08:39 on dist-test-slave-ltqz"
server_key: "788313845655fa83e613f0ae40d22654"
server_key_iv: "d5db1c6ed33567df6692971aa1c33bcf"
server_key_version: "encryptionkey@0"
I20260927 07:08:39.808487 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:39.811885 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:39.814231 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:39.814399 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:39.818756 10163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46255
I20260927 07:08:39.818787 10230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46255 every 8 connection(s)
I20260927 07:08:39.819120 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:39.820179 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:39.822917 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b: Bootstrap starting.
I20260927 07:08:39.823542 10231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:39.823835 10231 log.cc:824] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:39.824513 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b: No bootstrap required, opened a new log
I20260927 07:08:39.826074 10231 raft_consensus.cc:348] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260927 07:08:39.826197 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10163
I20260927 07:08:39.826231 10231 raft_consensus.cc:374] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:39.826300 10231 raft_consensus.cc:729] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f969192be2a2471198c1975e81f5aa0b, State: Initialized, Role: FOLLOWER
I20260927 07:08:39.826323 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:39.826454 10231 consensus_queue.cc:260] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260927 07:08:39.826495 21513 external_mini_cluster.cc:1428] Setting key 52a939ae7c7fd0a9cc39da846af80c7e
I20260927 07:08:39.826532 10231 raft_consensus.cc:388] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:39.826584 10231 raft_consensus.cc:482] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:39.826634 10231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:39.827757 10231 raft_consensus.cc:504] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260927 07:08:39.827864 10231 leader_election.cc:302] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f969192be2a2471198c1975e81f5aa0b; no voters:
I20260927 07:08:39.828015 10231 leader_election.cc:288] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:39.828133 10236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:39.828294 10231 sys_catalog.cc:564] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:39.828320 10236 raft_consensus.cc:686] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [term 1 LEADER]: Becoming Leader. State: Replica: f969192be2a2471198c1975e81f5aa0b, State: Running, Role: LEADER
I20260927 07:08:39.828513 10236 consensus_queue.cc:237] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } }
I20260927 07:08:39.828899 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:39.827018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:53695 (local address 127.0.0.1:46255)
0927 07:08:39.827212 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:39.827224 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:39.827249 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:08:39.827337 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:39.827342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:39.827410 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:39.827531 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:39.828260 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:39.828264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:39.828264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:39.828268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:39.828318 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:39.828335 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:39.828395 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:39.828637 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 07:08:39.828683 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260927 07:08:39.828882 10237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f969192be2a2471198c1975e81f5aa0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } } }
I20260927 07:08:39.829196 10237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:39.828994 10238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f969192be2a2471198c1975e81f5aa0b. Latest consensus state: current_term: 1 leader_uuid: "f969192be2a2471198c1975e81f5aa0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f969192be2a2471198c1975e81f5aa0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46255 } } }
I20260927 07:08:39.829506 10238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:39.829684 10245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:39.830042 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:39.830322 10245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:39.831980 10245 catalog_manager.cc:1347] Generated new cluster ID: 05703bf5a39c42368c807b1dc0a31d0a
I20260927 07:08:39.832031 10245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:39.832754 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:39.831584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:50205 (local address 127.0.0.1:46255)
0927 07:08:39.831625 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:39.831627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:39.831779 (+ 152us) server_negotiation.cc:407] Connection header received
0927 07:08:39.831831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:39.831831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:39.831837 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:39.831893 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:39.832529 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:39.832530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:39.832530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:39.832534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:39.832549 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:39.832554 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:39.832615 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:39.832684 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 07:08:39.832696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:08:39.843564 10245 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:39.843752 10245 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:39.848301 10245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f969192be2a2471198c1975e81f5aa0b: Generated new TSK 0
I20260927 07:08:39.848544 10245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:39.854861 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10163
2026-09-27T07:08:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:39 dist-test-slave-ltqz krb5kdc[10262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:39 dist-test-slave-ltqz krb5kdc[10262](info): set up 2 sockets
Sep 27 07:08:39 dist-test-slave-ltqz krb5kdc[10262](info): commencing operation
krb5kdc: starting...
W20260927 07:08:40.899819 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:40 dist-test-slave-ltqz krb5kdc[10262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:41.016609 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:41.098894 10275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:41.099164 10275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:41.099205 10275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:41.100467 10275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:41.100545 10275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:41.100570 10275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:41.100592 10275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:41.100621 10275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:41.102149 10275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:41.102455 10275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:41.102694 10275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:41.107868 10275 server_base.cc:1034] running on GCE node
W20260927 07:08:41.107767 10283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:41.107786 10281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:41.107810 10280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:41.108407 10275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:41.108700 10275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:41.109874 10275 hybrid_clock.cc:648] HybridClock initialized: now 1790492921109864 us; error 36 us; skew 500 ppm
Sep 27 07:08:41 dist-test-slave-ltqz krb5kdc[10262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:41.118981 10275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:41.119550 10275 webserver.cc:460] Webserver started at http://127.0.0.1:38929/ using document root <none> and password file <none>
I20260927 07:08:41.119799 10275 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:41.119874 10275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:41.120023 10275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:41.121280 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "439f071f87664db2b921ba076a59b2e3"
format_stamp: "Formatted at 2026-09-27 07:08:41 on dist-test-slave-ltqz"
server_key: "656c1c9d476bd391a97693380920e96f"
server_key_iv: "2e96daf2f60e31ab1480f6167de6932d"
server_key_version: "encryptionkey@0"
I20260927 07:08:41.121680 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "439f071f87664db2b921ba076a59b2e3"
format_stamp: "Formatted at 2026-09-27 07:08:41 on dist-test-slave-ltqz"
server_key: "656c1c9d476bd391a97693380920e96f"
server_key_iv: "2e96daf2f60e31ab1480f6167de6932d"
server_key_version: "encryptionkey@0"
I20260927 07:08:41.123143 10275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:41.123980 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:41.124184 10275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:41.124279 10275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "439f071f87664db2b921ba076a59b2e3"
format_stamp: "Formatted at 2026-09-27 07:08:41 on dist-test-slave-ltqz"
server_key: "656c1c9d476bd391a97693380920e96f"
server_key_iv: "2e96daf2f60e31ab1480f6167de6932d"
server_key_version: "encryptionkey@0"
I20260927 07:08:41.124358 10275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:41.150818 10275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:41.152989 10275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:41.153137 10275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:41.157136 10275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45139
I20260927 07:08:41.157176 10342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45139 every 8 connection(s)
I20260927 07:08:41.157552 10275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:41.158412 10343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:41.160776 10343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3: Bootstrap starting.
I20260927 07:08:41.161360 10343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:41.161666 10343 log.cc:824] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:41.161834 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10275
I20260927 07:08:41.161958 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:41.162133 21513 external_mini_cluster.cc:1428] Setting key 4f4636b76d41f9bb835cb912230ac345
I20260927 07:08:41.162551 10343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3: No bootstrap required, opened a new log
I20260927 07:08:41.164860 10343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 07:08:41.164994 10343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:41.165015 10343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439f071f87664db2b921ba076a59b2e3, State: Initialized, Role: FOLLOWER
I20260927 07:08:41.165148 10343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 07:08:41.165205 10343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:41.165253 10343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:41.165315 10343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:41.165876 10343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 07:08:41.165963 10343 leader_election.cc:302] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439f071f87664db2b921ba076a59b2e3; no voters:
I20260927 07:08:41.166163 10343 leader_election.cc:288] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:41.166184 10348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:41.166293 10348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [term 1 LEADER]: Becoming Leader. State: Replica: 439f071f87664db2b921ba076a59b2e3, State: Running, Role: LEADER
I20260927 07:08:41.166422 10343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:41.166409 10348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 07:08:41.166903 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439f071f87664db2b921ba076a59b2e3. Latest consensus state: current_term: 1 leader_uuid: "439f071f87664db2b921ba076a59b2e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 07:08:41.167013 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:41.167232 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439f071f87664db2b921ba076a59b2e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439f071f87664db2b921ba076a59b2e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 07:08:41.167546 10356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:41.168093 10356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:41.169190 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:41.170130 10356 catalog_manager.cc:1347] Generated new cluster ID: 0de0452ad8e344eb8d2fd716d6953844
I20260927 07:08:41.170181 10356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:41 dist-test-slave-ltqz krb5kdc[10262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:08:41.174818 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:41.162646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:53079 (local address 127.0.0.1:45139)
0927 07:08:41.162832 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:41.162841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:41.162875 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:08:41.163788 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:41.163795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:41.163845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:41.163962 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:41.164791 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:41.166222 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:41.167241 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:41.167895 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:41.168009 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:41.170919 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:41.170923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:41.170924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:41.170937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:41.172987 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:41.173419 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:41.173422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:41.173423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:41.173540 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:41.173772 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:41.173773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:41.173774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:41.174096 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:41.174267 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:41.174361 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 07:08:41.174466 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260927 07:08:41.175915 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:41.179298 10356 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:41.180023 10356 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:41.184625 10347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:41.177441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:46505 (local address 127.0.0.1:45139)
0927 07:08:41.177477 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:41.177479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:41.177488 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:08:41.177511 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:41.177512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:41.177531 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:41.177595 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:41.178214 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:41.178986 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:41.180034 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:41.180348 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:41.180421 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:41.181830 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:41.181831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:41.181831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:41.181838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:41.183429 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:41.183825 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:41.183826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:41.183827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:41.183885 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:41.184111 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:41.184112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:41.184112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:41.184240 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:41.184348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:41.184463 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 07:08:41.184534 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:08:41.185314 10356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439f071f87664db2b921ba076a59b2e3: Generated new TSK 0
I20260927 07:08:41.185482 10356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:41.190099 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10275
W20260927 07:08:41.194458 10260 connection.cc:535] client connection to 127.0.0.1:45139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45139): Connection reset by peer (error 104)
2026-09-27T07:08:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:41 dist-test-slave-ltqz krb5kdc[10374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:41 dist-test-slave-ltqz krb5kdc[10374](info): set up 2 sockets
Sep 27 07:08:41 dist-test-slave-ltqz krb5kdc[10374](info): commencing operation
krb5kdc: starting...
W20260927 07:08:42.235625 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:42 dist-test-slave-ltqz krb5kdc[10374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:42.352561 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:42.443882 10387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:42.444156 10387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:42.444197 10387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:42.445401 10387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:42.445482 10387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:42.445508 10387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:42.445545 10387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:42.445595 10387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:42.447220 10387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:42.447549 10387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:42.447958 10387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:42.452903 10393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:42.452896 10395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:42.452932 10387 server_base.cc:1034] running on GCE node
W20260927 07:08:42.452896 10392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:42.453351 10387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:42.453653 10387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:42.454807 10387 hybrid_clock.cc:648] HybridClock initialized: now 1790492922454796 us; error 31 us; skew 500 ppm
Sep 27 07:08:42 dist-test-slave-ltqz krb5kdc[10374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:42.457273 10387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:42.457650 10387 webserver.cc:460] Webserver started at http://127.0.0.1:42543/ using document root <none> and password file <none>
I20260927 07:08:42.457828 10387 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:42.457866 10387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:42.457957 10387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:42.458973 10387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "6e35db7a64554f968d2d92a28cdb6fcc"
format_stamp: "Formatted at 2026-09-27 07:08:42 on dist-test-slave-ltqz"
server_key: "7a2e9c393b62c23d28320dbcd797416a"
server_key_iv: "b60201e265fe354b417d0ffbdce50ebb"
server_key_version: "encryptionkey@0"
I20260927 07:08:42.459268 10387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "6e35db7a64554f968d2d92a28cdb6fcc"
format_stamp: "Formatted at 2026-09-27 07:08:42 on dist-test-slave-ltqz"
server_key: "7a2e9c393b62c23d28320dbcd797416a"
server_key_iv: "b60201e265fe354b417d0ffbdce50ebb"
server_key_version: "encryptionkey@0"
I20260927 07:08:42.460588 10387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:42.461252 10402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:42.461424 10387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:42.461535 10387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "6e35db7a64554f968d2d92a28cdb6fcc"
format_stamp: "Formatted at 2026-09-27 07:08:42 on dist-test-slave-ltqz"
server_key: "7a2e9c393b62c23d28320dbcd797416a"
server_key_iv: "b60201e265fe354b417d0ffbdce50ebb"
server_key_version: "encryptionkey@0"
I20260927 07:08:42.461611 10387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:42.470849 10387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:42.472617 10387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:42.472769 10387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:42.476538 10387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37647
I20260927 07:08:42.476574 10454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37647 every 8 connection(s)
I20260927 07:08:42.476927 10387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:42.477574 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10387
I20260927 07:08:42.477697 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:42.477887 21513 external_mini_cluster.cc:1428] Setting key 5004b6131148e81702182796fdbd6b40
I20260927 07:08:42.477898 10455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:42.481040 10455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc: Bootstrap starting.
I20260927 07:08:42.481660 10455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:42.481981 10455 log.cc:824] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:42.482764 10455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc: No bootstrap required, opened a new log
I20260927 07:08:42.483291 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:42.478383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:46273 (local address 127.0.0.1:37647)
0927 07:08:42.478556 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:42.478564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:42.478588 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:08:42.478990 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:42.478996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:42.479042 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:42.479149 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:42.479422 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:42.480300 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:42.481373 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:42.481810 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:42.481887 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:42.482636 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:42.482640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:42.482640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:42.482790 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:42.482850 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:42.482865 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:42.482939 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:42.483035 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:08:42.483077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260927 07:08:42.484439 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:42.485116 10455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 07:08:42.485296 10455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:42.485330 10455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e35db7a64554f968d2d92a28cdb6fcc, State: Initialized, Role: FOLLOWER
I20260927 07:08:42.485442 10455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 07:08:42.485551 10455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:42.485639 10455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:42.485697 10455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:42.486464 10455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 07:08:42.486546 10455 leader_election.cc:302] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e35db7a64554f968d2d92a28cdb6fcc; no voters:
I20260927 07:08:42.486719 10455 leader_election.cc:288] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:42.486750 10465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:42.486860 10465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [term 1 LEADER]: Becoming Leader. State: Replica: 6e35db7a64554f968d2d92a28cdb6fcc, State: Running, Role: LEADER
I20260927 07:08:42.486955 10455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:42.487010 10465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } }
I20260927 07:08:42.487397 10465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e35db7a64554f968d2d92a28cdb6fcc. Latest consensus state: current_term: 1 leader_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } } }
I20260927 07:08:42.487643 10465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:42.487560 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e35db7a64554f968d2d92a28cdb6fcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37647 } } }
I20260927 07:08:42.488095 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:42.488268 10474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:42.488792 10474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:42.490116 10474 catalog_manager.cc:1347] Generated new cluster ID: 8f0a7216567340608180ea8234f5b891
I20260927 07:08:42.490172 10474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:42.496209 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:42.485926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:39541 (local address 127.0.0.1:37647)
0927 07:08:42.485960 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:42.485962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:42.486070 (+ 108us) server_negotiation.cc:407] Connection header received
0927 07:08:42.486135 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:42.486136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:42.486143 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:42.486189 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:42.486324 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:42.486929 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:42.487961 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:42.488276 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:42.488339 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:42.495942 (+ 7603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:42.495943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:42.495944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:42.495949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:42.495964 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:42.495969 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:42.496039 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:42.496124 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 07:08:42.496135 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:42.502853 10474 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:42.503331 10474 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:42.510741 10474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e35db7a64554f968d2d92a28cdb6fcc: Generated new TSK 0
I20260927 07:08:42.510946 10474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:42.516793 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10387
W20260927 07:08:42.521188 10372 connection.cc:535] client connection to 127.0.0.1:37647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37647): Connection reset by peer (error 104)
2026-09-27T07:08:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:42 dist-test-slave-ltqz krb5kdc[10486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:42 dist-test-slave-ltqz krb5kdc[10486](info): set up 2 sockets
Sep 27 07:08:42 dist-test-slave-ltqz krb5kdc[10486](info): commencing operation
krb5kdc: starting...
W20260927 07:08:43.551555 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:43 dist-test-slave-ltqz krb5kdc[10486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:08:43.667840 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:43.747558 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:43.747816 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:43.747856 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:43.748998 10499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:43.749084 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:43.749109 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:43.749137 10499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:43.749169 10499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:43.750643 10499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:43.750926 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:43.751160 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:43.755889 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:43.755887 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:43.755884 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:43.756124 10499 server_base.cc:1034] running on GCE node
I20260927 07:08:43.756361 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:43.756659 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:43.757854 10499 hybrid_clock.cc:648] HybridClock initialized: now 1790492923757845 us; error 35 us; skew 500 ppm
Sep 27 07:08:43 dist-test-slave-ltqz krb5kdc[10486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:43.760389 10499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:08:43.760866 10499 webserver.cc:460] Webserver started at http://127.0.0.1:33509/ using document root <none> and password file <none>
I20260927 07:08:43.761096 10499 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:43.761164 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:43.761315 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:43.762370 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "01c3a9fc5a7442f4b215751539fb2229"
format_stamp: "Formatted at 2026-09-27 07:08:43 on dist-test-slave-ltqz"
server_key: "47b350a9392563271466f64edd2b841a"
server_key_iv: "44885e050cdc4e5efc97b7b28b55181c"
server_key_version: "encryptionkey@0"
I20260927 07:08:43.762737 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "01c3a9fc5a7442f4b215751539fb2229"
format_stamp: "Formatted at 2026-09-27 07:08:43 on dist-test-slave-ltqz"
server_key: "47b350a9392563271466f64edd2b841a"
server_key_iv: "44885e050cdc4e5efc97b7b28b55181c"
server_key_version: "encryptionkey@0"
I20260927 07:08:43.764183 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:43.764931 10514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:43.765102 10499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:43.765200 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "01c3a9fc5a7442f4b215751539fb2229"
format_stamp: "Formatted at 2026-09-27 07:08:43 on dist-test-slave-ltqz"
server_key: "47b350a9392563271466f64edd2b841a"
server_key_iv: "44885e050cdc4e5efc97b7b28b55181c"
server_key_version: "encryptionkey@0"
I20260927 07:08:43.765282 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:43.768601 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:43.770680 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:43.770856 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:43.774964 10499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46661
I20260927 07:08:43.774993 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46661 every 8 connection(s)
I20260927 07:08:43.775321 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:43.776290 10567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:43.778378 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229: Bootstrap starting.
I20260927 07:08:43.778894 10567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:43.779115 10567 log.cc:824] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:43.779793 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229: No bootstrap required, opened a new log
I20260927 07:08:43.781268 10567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260927 07:08:43.781397 10567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:43.781427 10567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c3a9fc5a7442f4b215751539fb2229, State: Initialized, Role: FOLLOWER
I20260927 07:08:43.781533 10567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260927 07:08:43.781581 10567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:43.781601 10567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:43.781626 10567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:43.782083 10567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260927 07:08:43.782164 10567 leader_election.cc:302] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c3a9fc5a7442f4b215751539fb2229; no voters:
I20260927 07:08:43.782341 10567 leader_election.cc:288] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:43.782373 10570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:43.782491 10570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [term 1 LEADER]: Becoming Leader. State: Replica: 01c3a9fc5a7442f4b215751539fb2229, State: Running, Role: LEADER
I20260927 07:08:43.782452 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10499
I20260927 07:08:43.782588 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:43.782618 10567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:43.782598 10570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260927 07:08:43.782776 21513 external_mini_cluster.cc:1428] Setting key 6d997a83130f490d3e4cdc64f701ae30
I20260927 07:08:43.782932 10570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c3a9fc5a7442f4b215751539fb2229. Latest consensus state: current_term: 1 leader_uuid: "01c3a9fc5a7442f4b215751539fb2229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
I20260927 07:08:43.783016 10570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:43.783074 10571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c3a9fc5a7442f4b215751539fb2229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3a9fc5a7442f4b215751539fb2229" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
I20260927 07:08:43.783192 10571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:43.783349 10574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:43.783888 10574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:43.786057 10574 catalog_manager.cc:1347] Generated new cluster ID: 3573656341c04672a3d3992949e68b7b
I20260927 07:08:43.786111 10574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:43.786247 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:43.783258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:54963 (local address 127.0.0.1:46661)
0927 07:08:43.783985 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:43.784006 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:43.784683 (+ 677us) server_negotiation.cc:407] Connection header received
0927 07:08:43.784857 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:43.784865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:43.784894 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:43.785038 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:43.785756 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:43.785760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:43.785760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:43.785765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:43.785806 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:43.785821 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:43.785894 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:43.786027 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 07:08:43.786073 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":80,"threads_started":1}
I20260927 07:08:43.787165 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:43.789810 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:43.788665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:58175 (local address 127.0.0.1:46661)
0927 07:08:43.788725 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:43.788727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:43.788918 (+ 191us) server_negotiation.cc:407] Connection header received
0927 07:08:43.788989 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:43.788990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:43.788995 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:43.789050 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:43.789618 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:43.789619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:43.789619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:43.789623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:43.789636 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:43.789640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:43.789689 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:43.789744 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 07:08:43.789755 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:08:43.794776 10574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:43.794937 10574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:43.799484 10574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c3a9fc5a7442f4b215751539fb2229: Generated new TSK 0
I20260927 07:08:43.799680 10574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:43.804275 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10499
2026-09-27T07:08:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:43 dist-test-slave-ltqz krb5kdc[10598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:43 dist-test-slave-ltqz krb5kdc[10598](info): set up 2 sockets
Sep 27 07:08:43 dist-test-slave-ltqz krb5kdc[10598](info): commencing operation
krb5kdc: starting...
W20260927 07:08:44.839645 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:44 dist-test-slave-ltqz krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:44.957613 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36887
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:45.039064 10611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:45.039333 10611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:45.039376 10611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:45.040604 10611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:45.040685 10611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:45.040710 10611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:45.040733 10611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:45.040764 10611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:45.042332 10611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:36887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:45.042634 10611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:45.042870 10611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:45.047820 10616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:45.047791 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:45.047853 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:45.047991 10611 server_base.cc:1034] running on GCE node
I20260927 07:08:45.048297 10611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:45.048606 10611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:45.049783 10611 hybrid_clock.cc:648] HybridClock initialized: now 1790492925049777 us; error 40 us; skew 500 ppm
Sep 27 07:08:45 dist-test-slave-ltqz krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492925, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:45.052279 10611 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:45.052708 10611 webserver.cc:460] Webserver started at http://127.21.2.126:41355/ using document root <none> and password file <none>
I20260927 07:08:45.052949 10611 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:45.053020 10611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:45.053174 10611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:45.054225 10611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "183f1baa0ae348599b61547f9e1ac853"
format_stamp: "Formatted at 2026-09-27 07:08:45 on dist-test-slave-ltqz"
server_key: "30d44c0bb636ca81a63eddc1a3efd519"
server_key_iv: "2650b19764d54afc885983cc14b1f6e5"
server_key_version: "encryptionkey@0"
I20260927 07:08:45.054605 10611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "183f1baa0ae348599b61547f9e1ac853"
format_stamp: "Formatted at 2026-09-27 07:08:45 on dist-test-slave-ltqz"
server_key: "30d44c0bb636ca81a63eddc1a3efd519"
server_key_iv: "2650b19764d54afc885983cc14b1f6e5"
server_key_version: "encryptionkey@0"
I20260927 07:08:45.056000 10611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:45.056718 10626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:45.056887 10611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:45.056986 10611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "183f1baa0ae348599b61547f9e1ac853"
format_stamp: "Formatted at 2026-09-27 07:08:45 on dist-test-slave-ltqz"
server_key: "30d44c0bb636ca81a63eddc1a3efd519"
server_key_iv: "2650b19764d54afc885983cc14b1f6e5"
server_key_version: "encryptionkey@0"
I20260927 07:08:45.057068 10611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:45.076251 10611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:45.078408 10611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:45.078591 10611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:45.082798 10611 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:36887
I20260927 07:08:45.082834 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:36887 every 8 connection(s)
I20260927 07:08:45.083218 10611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:45.084125 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:45.086354 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853: Bootstrap starting.
I20260927 07:08:45.086952 10679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:45.087242 10679 log.cc:824] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:45.087956 10679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853: No bootstrap required, opened a new log
I20260927 07:08:45.089533 10679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } }
I20260927 07:08:45.089676 10679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:45.089699 10679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183f1baa0ae348599b61547f9e1ac853, State: Initialized, Role: FOLLOWER
I20260927 07:08:45.089856 10679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } }
I20260927 07:08:45.089929 10679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:45.089951 10679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:45.090008 10679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:45.090874 10679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } }
I20260927 07:08:45.090974 10679 leader_election.cc:302] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 183f1baa0ae348599b61547f9e1ac853; no voters:
I20260927 07:08:45.091171 10679 leader_election.cc:288] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:45.091295 10682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:45.091539 10679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:45.091588 10682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [term 1 LEADER]: Becoming Leader. State: Replica: 183f1baa0ae348599b61547f9e1ac853, State: Running, Role: LEADER
I20260927 07:08:45.091701 10682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } }
I20260927 07:08:45.092109 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "183f1baa0ae348599b61547f9e1ac853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } } }
I20260927 07:08:45.092202 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:45.092298 10684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 183f1baa0ae348599b61547f9e1ac853. Latest consensus state: current_term: 1 leader_uuid: "183f1baa0ae348599b61547f9e1ac853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183f1baa0ae348599b61547f9e1ac853" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36887 } } }
I20260927 07:08:45.092465 10684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:45.092743 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10611
I20260927 07:08:45.092790 10691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:45.092868 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:45.093080 21513 external_mini_cluster.cc:1428] Setting key 1afe66219c1ce0ab8c14f7eb89c5ff33
I20260927 07:08:45.093878 10691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:45.095818 10691 catalog_manager.cc:1347] Generated new cluster ID: 2fae39c69e5b45049ec35513baa56e2e
I20260927 07:08:45.095884 10691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:45 dist-test-slave-ltqz krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:45.106344 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:45.094155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41877 (local address 127.21.2.126:36887)
0927 07:08:45.094382 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:45.094391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:45.094427 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:08:45.094752 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:45.094761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:45.094826 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:45.094994 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:45.096002 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:45.097414 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:45.098449 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:45.099105 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:45.099228 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:45.102382 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:45.102387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:45.102387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:45.102402 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:45.104570 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:45.105028 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:45.105031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:45.105031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:45.105159 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:45.105394 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:45.105395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:45.105395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:45.105713 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:45.105883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:45.105996 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 07:08:45.106111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260927 07:08:45.107450 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:45.116956 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:45.108957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44443 (local address 127.21.2.126:36887)
0927 07:08:45.108996 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:45.108998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:45.109129 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:08:45.109188 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:45.109189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:45.109214 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:45.109266 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:45.109897 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:45.110743 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:45.111837 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:45.112155 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:45.112303 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:45.113937 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:45.113938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:45.113939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:45.113946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:45.115577 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:45.116010 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:45.116011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:45.116011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:45.116065 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:45.116288 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:45.116288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:45.116288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:45.116499 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:45.116616 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:45.116787 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 07:08:45.116854 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:08:45.125216 10691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:45.125754 10691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:45.129490 10691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 183f1baa0ae348599b61547f9e1ac853: Generated new TSK 0
I20260927 07:08:45.129684 10691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:45.136507 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10611
W20260927 07:08:45.140944 10596 connection.cc:535] client connection to 127.21.2.126:36887 recv error: Network error: failed to read from TLS socket (remote: 127.21.2.126:36887): Connection reset by peer (error 104)
2026-09-27T07:08:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:45 dist-test-slave-ltqz krb5kdc[10710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:45 dist-test-slave-ltqz krb5kdc[10710](info): set up 2 sockets
Sep 27 07:08:45 dist-test-slave-ltqz krb5kdc[10710](info): commencing operation
krb5kdc: starting...
W20260927 07:08:46.191573 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:46 dist-test-slave-ltqz krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:46.305943 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46229
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:46.386272 10723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:46.386498 10723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:46.386538 10723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:46.387744 10723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:46.387804 10723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:46.387830 10723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:46.387864 10723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:46.387892 10723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:46.389315 10723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:46229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:46.389567 10723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:46.389794 10723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:46.394596 10731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:46.394605 10728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:46.394806 10723 server_base.cc:1034] running on GCE node
W20260927 07:08:46.394840 10729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:46.395141 10723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:46.395517 10723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:46.396677 10723 hybrid_clock.cc:648] HybridClock initialized: now 1790492926396654 us; error 38 us; skew 500 ppm
Sep 27 07:08:46 dist-test-slave-ltqz krb5kdc[10710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492926, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:46.402639 10723 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:46.403095 10723 webserver.cc:460] Webserver started at http://127.21.2.126:45943/ using document root <none> and password file <none>
I20260927 07:08:46.403350 10723 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:46.403473 10723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:46.403633 10723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:46.404744 10723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7"
format_stamp: "Formatted at 2026-09-27 07:08:46 on dist-test-slave-ltqz"
server_key: "827a7dc43ad7249f8107240464cc9b9e"
server_key_iv: "59264a521aa405a4f071da8874a07e3e"
server_key_version: "encryptionkey@0"
I20260927 07:08:46.405129 10723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7"
format_stamp: "Formatted at 2026-09-27 07:08:46 on dist-test-slave-ltqz"
server_key: "827a7dc43ad7249f8107240464cc9b9e"
server_key_iv: "59264a521aa405a4f071da8874a07e3e"
server_key_version: "encryptionkey@0"
I20260927 07:08:46.406597 10723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:46.407322 10738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:46.407521 10723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:46.407624 10723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7"
format_stamp: "Formatted at 2026-09-27 07:08:46 on dist-test-slave-ltqz"
server_key: "827a7dc43ad7249f8107240464cc9b9e"
server_key_iv: "59264a521aa405a4f071da8874a07e3e"
server_key_version: "encryptionkey@0"
I20260927 07:08:46.407713 10723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:46.427191 10723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:46.429080 10723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:46.429251 10723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:46.433380 10723 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:46229
I20260927 07:08:46.433416 10790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:46229 every 8 connection(s)
I20260927 07:08:46.433799 10723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:46.434787 10791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:46.437043 10791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: Bootstrap starting.
I20260927 07:08:46.437566 10791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:46.437793 10791 log.cc:824] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:46.438453 10791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: No bootstrap required, opened a new log
I20260927 07:08:46.440039 10791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } }
I20260927 07:08:46.440171 10791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:46.440193 10791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f9fd04f35e4ba19e75ec22fb0ccfc7, State: Initialized, Role: FOLLOWER
I20260927 07:08:46.440323 10791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } }
I20260927 07:08:46.440407 10791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:46.440429 10791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:46.440457 10791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:46.440905 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10723
I20260927 07:08:46.441039 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:46.441221 21513 external_mini_cluster.cc:1428] Setting key a85057ee10fd0eb5ab2d0e2e4ee6b1b4
I20260927 07:08:46.441372 10791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } }
I20260927 07:08:46.441484 10791 leader_election.cc:302] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58f9fd04f35e4ba19e75ec22fb0ccfc7; no voters:
I20260927 07:08:46.441659 10791 leader_election.cc:288] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:46.441821 10796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:46.441910 10791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:46.442029 10796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 58f9fd04f35e4ba19e75ec22fb0ccfc7, State: Running, Role: LEADER
I20260927 07:08:46.442175 10796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } }
I20260927 07:08:46.442512 10796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f9fd04f35e4ba19e75ec22fb0ccfc7. Latest consensus state: current_term: 1 leader_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } } }
I20260927 07:08:46.442749 10796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:46.442667 10794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f9fd04f35e4ba19e75ec22fb0ccfc7" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 46229 } } }
I20260927 07:08:46.443015 10794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7 [sys.catalog]: This master's current role is: LEADER
W20260927 07:08:46.443869 10812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:46.443943 10812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:46.443995 10805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:46.444201 10805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:46.445670 10805 catalog_manager.cc:1347] Generated new cluster ID: 9dc01ce4895a4263a0437b44653edbdc
I20260927 07:08:46.445726 10805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:46.447078 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:46.441720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37661 (local address 127.21.2.126:46229)
0927 07:08:46.442019 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:46.442030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:46.442065 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:08:46.442179 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:46.442186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:46.442243 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:46.442406 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:46.442693 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:46.444068 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:46.445013 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:46.445621 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:46.445733 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:46.446443 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:46.446446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:46.446447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:46.446645 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:46.446681 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:46.446695 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:46.446775 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:46.446854 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 07:08:46.446880 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":64,"threads_started":1}
I20260927 07:08:46.448236 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:46.457325 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:46.449551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57131 (local address 127.21.2.126:46229)
0927 07:08:46.449592 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:46.449594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:46.454462 (+ 4868us) server_negotiation.cc:407] Connection header received
0927 07:08:46.454537 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:46.454538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:46.454549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:46.454606 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:46.454748 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:46.455237 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:46.455993 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:46.456213 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:46.456306 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:46.456972 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:46.456973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:46.456973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:46.456978 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:46.456992 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:46.456996 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:46.457085 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:46.457225 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:08:46.457241 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:08:46.468508 10805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:46.469026 10805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:46.475505 10805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58f9fd04f35e4ba19e75ec22fb0ccfc7: Generated new TSK 0
I20260927 07:08:46.475688 10805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:46.481109 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10723
W20260927 07:08:46.485682 10708 connection.cc:535] client connection to 127.21.2.126:46229 recv error: Network error: failed to read from TLS socket (remote: 127.21.2.126:46229): Connection reset by peer (error 104)
2026-09-27T07:08:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:46 dist-test-slave-ltqz krb5kdc[10822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:46 dist-test-slave-ltqz krb5kdc[10822](info): set up 2 sockets
Sep 27 07:08:46 dist-test-slave-ltqz krb5kdc[10822](info): commencing operation
krb5kdc: starting...
W20260927 07:08:47.515472 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:47 dist-test-slave-ltqz krb5kdc[10822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:47.634877 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36775
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:47.714653 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:47.714898 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:47.714936 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:47.716110 10835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:47.716185 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:47.716209 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:47.716231 10835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:47.716282 10835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:47.717741 10835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:47.718024 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:47.718254 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:47.722968 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:47.722980 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:47.722988 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:47.723004 10835 server_base.cc:1034] running on GCE node
I20260927 07:08:47.723487 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:47.723779 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:47.724965 10835 hybrid_clock.cc:648] HybridClock initialized: now 1790492927724963 us; error 42 us; skew 500 ppm
Sep 27 07:08:47 dist-test-slave-ltqz krb5kdc[10822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492927, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:47.727924 10835 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:47.728410 10835 webserver.cc:460] Webserver started at http://127.21.2.126:42081/ using document root <none> and password file <none>
I20260927 07:08:47.728648 10835 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:47.728698 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:47.728856 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:47.729972 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "aae786b4f5b14349a1961e088334cd50"
format_stamp: "Formatted at 2026-09-27 07:08:47 on dist-test-slave-ltqz"
server_key: "bff5070129644e76df1a62f46eed2be6"
server_key_iv: "70ee98d46d5a0bd4263a878d9f237929"
server_key_version: "encryptionkey@0"
I20260927 07:08:47.730373 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "aae786b4f5b14349a1961e088334cd50"
format_stamp: "Formatted at 2026-09-27 07:08:47 on dist-test-slave-ltqz"
server_key: "bff5070129644e76df1a62f46eed2be6"
server_key_iv: "70ee98d46d5a0bd4263a878d9f237929"
server_key_version: "encryptionkey@0"
I20260927 07:08:47.731868 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:47.732702 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:47.732873 10835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:08:47.732975 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "aae786b4f5b14349a1961e088334cd50"
format_stamp: "Formatted at 2026-09-27 07:08:47 on dist-test-slave-ltqz"
server_key: "bff5070129644e76df1a62f46eed2be6"
server_key_iv: "70ee98d46d5a0bd4263a878d9f237929"
server_key_version: "encryptionkey@0"
I20260927 07:08:47.733054 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:47.736693 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:47.738725 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:47.738893 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:47.742676 10835 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:36775
I20260927 07:08:47.742739 10902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:36775 every 8 connection(s)
I20260927 07:08:47.743084 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:47.744115 10903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:47.746263 10903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50: Bootstrap starting.
I20260927 07:08:47.746826 10903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:47.747099 10903 log.cc:824] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:47.747808 10903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50: No bootstrap required, opened a new log
I20260927 07:08:47.749349 10903 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } }
I20260927 07:08:47.749531 10903 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:47.749578 10903 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae786b4f5b14349a1961e088334cd50, State: Initialized, Role: FOLLOWER
I20260927 07:08:47.749619 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10835
I20260927 07:08:47.749734 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:47.749722 10903 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } }
I20260927 07:08:47.749794 10903 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:47.749840 10903 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:47.749887 10903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:47.749914 21513 external_mini_cluster.cc:1428] Setting key 95df2d2b034e645cf53048de44c701cc
I20260927 07:08:47.750833 10903 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } }
I20260927 07:08:47.750948 10903 leader_election.cc:302] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae786b4f5b14349a1961e088334cd50; no voters:
I20260927 07:08:47.751147 10903 leader_election.cc:288] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:47.751163 10908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:47.751318 10908 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [term 1 LEADER]: Becoming Leader. State: Replica: aae786b4f5b14349a1961e088334cd50, State: Running, Role: LEADER
I20260927 07:08:47.751464 10903 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:47.751448 10908 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } }
I20260927 07:08:47.751927 10909 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae786b4f5b14349a1961e088334cd50. Latest consensus state: current_term: 1 leader_uuid: "aae786b4f5b14349a1961e088334cd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } } }
I20260927 07:08:47.752089 10909 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:47.751961 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae786b4f5b14349a1961e088334cd50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae786b4f5b14349a1961e088334cd50" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 36775 } } }
I20260927 07:08:47.752211 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:47.752360 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:47.750417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59241 (local address 127.21.2.126:36775)
0927 07:08:47.750640 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:47.750648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:47.750672 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:08:47.750766 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:47.750772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:47.750794 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:47.750923 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:47.751738 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:47.751741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:47.751741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:47.751745 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:47.751789 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:47.751804 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:47.751864 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:47.752074 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 07:08:47.752117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
I20260927 07:08:47.752440 10917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:47.752974 10917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:47.753301 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:47.755875 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:47.754724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53121 (local address 127.21.2.126:36775)
0927 07:08:47.754781 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:47.754782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:47.754913 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:08:47.754967 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:47.754968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:47.754972 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:47.755016 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:47.755629 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:47.755630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:47.755630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:47.755634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:47.755648 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:47.755653 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:47.755683 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:47.755805 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:08:47.755819 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:08:47.756114 10925 mvcc.cc:205] Tried to move back new op lower bound from 7333859032080097280 to 7333859032070721536. Current Snapshot: MvccSnapshot[applied={T|T < 7333859032080097280}]
I20260927 07:08:47.756500 10917 catalog_manager.cc:1347] Generated new cluster ID: 44498c795a69456096e44850f0176c31
I20260927 07:08:47.756547 10917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:47.764762 10917 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:47.764922 10917 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:47.769918 10917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae786b4f5b14349a1961e088334cd50: Generated new TSK 0
I20260927 07:08:47.770088 10917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:47.774402 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10835
2026-09-27T07:08:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:47 dist-test-slave-ltqz krb5kdc[10934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:47 dist-test-slave-ltqz krb5kdc[10934](info): set up 2 sockets
Sep 27 07:08:47 dist-test-slave-ltqz krb5kdc[10934](info): commencing operation
krb5kdc: starting...
W20260927 07:08:48.795490 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:48 dist-test-slave-ltqz krb5kdc[10934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:48.910708 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45519
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:48.991716 10947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:48.991935 10947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:48.991957 10947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:48.993057 10947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:48.993095 10947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:48.993109 10947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:48.993122 10947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:48.993134 10947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:48.994549 10947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:48.994756 10947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:48.994936 10947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:49.000349 10953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:49.000345 10955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:49.000345 10952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:49.000574 10947 server_base.cc:1034] running on GCE node
I20260927 07:08:49.000873 10947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:49.001232 10947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:49.002404 10947 hybrid_clock.cc:648] HybridClock initialized: now 1790492929002388 us; error 26 us; skew 500 ppm
Sep 27 07:08:49 dist-test-slave-ltqz krb5kdc[10934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:49.005108 10947 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:49.005566 10947 webserver.cc:460] Webserver started at http://127.21.2.126:42519/ using document root <none> and password file <none>
I20260927 07:08:49.005833 10947 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:49.005903 10947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:49.006064 10947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:49.007154 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "368a2e470e514e3d8aa6a333aa9bd63c"
format_stamp: "Formatted at 2026-09-27 07:08:49 on dist-test-slave-ltqz"
server_key: "37ab409cc301c9fcdba6bd26e375bed0"
server_key_iv: "4d409e3367f2fe7296903241807bc651"
server_key_version: "encryptionkey@0"
I20260927 07:08:49.007551 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "368a2e470e514e3d8aa6a333aa9bd63c"
format_stamp: "Formatted at 2026-09-27 07:08:49 on dist-test-slave-ltqz"
server_key: "37ab409cc301c9fcdba6bd26e375bed0"
server_key_iv: "4d409e3367f2fe7296903241807bc651"
server_key_version: "encryptionkey@0"
I20260927 07:08:49.008924 10947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:49.009788 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:49.010143 10947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:49.010257 10947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "368a2e470e514e3d8aa6a333aa9bd63c"
format_stamp: "Formatted at 2026-09-27 07:08:49 on dist-test-slave-ltqz"
server_key: "37ab409cc301c9fcdba6bd26e375bed0"
server_key_iv: "4d409e3367f2fe7296903241807bc651"
server_key_version: "encryptionkey@0"
I20260927 07:08:49.010345 10947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:49.032008 10947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:49.034430 10947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:49.034602 10947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:49.038444 10947 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:45519
I20260927 07:08:49.038515 11014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:45519 every 8 connection(s)
I20260927 07:08:49.038872 10947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:49.039855 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:49.042187 11015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c: Bootstrap starting.
I20260927 07:08:49.042722 11015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:49.042959 11015 log.cc:824] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:49.043833 11015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c: No bootstrap required, opened a new log
I20260927 07:08:49.045374 11015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } }
I20260927 07:08:49.045516 11015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:49.045538 11015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368a2e470e514e3d8aa6a333aa9bd63c, State: Initialized, Role: FOLLOWER
I20260927 07:08:49.045673 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 10947
I20260927 07:08:49.045693 11015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } }
I20260927 07:08:49.045760 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:49.045771 11015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:49.045805 11015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:49.045835 11015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:49.045910 21513 external_mini_cluster.cc:1428] Setting key 1d816ab6e92be3d6f18c970cc95f94fa
I20260927 07:08:49.046886 11015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } }
I20260927 07:08:49.046981 11015 leader_election.cc:302] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368a2e470e514e3d8aa6a333aa9bd63c; no voters:
I20260927 07:08:49.047192 11015 leader_election.cc:288] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:49.047230 11020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:49.047461 11020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [term 1 LEADER]: Becoming Leader. State: Replica: 368a2e470e514e3d8aa6a333aa9bd63c, State: Running, Role: LEADER
I20260927 07:08:49.047501 11015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:49.047637 11020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } }
I20260927 07:08:49.048074 11021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } } }
I20260927 07:08:49.048157 11021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:49.048394 11022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 368a2e470e514e3d8aa6a333aa9bd63c. Latest consensus state: current_term: 1 leader_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a2e470e514e3d8aa6a333aa9bd63c" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 45519 } } }
I20260927 07:08:49.048504 11022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:49.048846 11027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:49.049394 11027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:49.050938 11027 catalog_manager.cc:1347] Generated new cluster ID: b929495f6cb04f7e978344185e9a46e4
I20260927 07:08:49.050993 11027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:08:49 dist-test-slave-ltqz krb5kdc[10934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:49.056859 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:49.046461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45839 (local address 127.21.2.126:45519)
0927 07:08:49.046697 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:49.046707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:49.046745 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:08:49.046841 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:49.046847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:49.046890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:49.047042 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:49.047936 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:49.049451 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:49.050532 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:49.051052 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:49.051174 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:49.053528 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:49.053532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:49.053532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:49.053544 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:49.055269 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:49.055670 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:49.055673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:49.055674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:49.055783 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:49.056001 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:49.056002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:49.056002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:49.056252 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:49.056432 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:49.056515 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:08:49.056655 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":117,"threads_started":1}
I20260927 07:08:49.057812 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:49.065428 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:49.059218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60631 (local address 127.21.2.126:45519)
0927 07:08:49.059253 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:49.059255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:49.059495 (+ 240us) server_negotiation.cc:407] Connection header received
0927 07:08:49.059567 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:49.059567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:49.059579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:49.059628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:49.060125 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:49.060606 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:49.061302 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:49.061517 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:49.061593 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:49.062776 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:49.062778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:49.062778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:49.062787 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:49.064297 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:49.064656 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:49.064657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:49.064658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:49.064714 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:49.064918 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:49.064919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:49.064919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:49.065034 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:49.065140 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:49.065244 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 07:08:49.065324 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:08:49.065882 11027 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:49.066601 11027 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:49.073891 11027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 368a2e470e514e3d8aa6a333aa9bd63c: Generated new TSK 0
I20260927 07:08:49.074118 11027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:49.081270 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 10947
2026-09-27T07:08:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:49 dist-test-slave-ltqz krb5kdc[11046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:49 dist-test-slave-ltqz krb5kdc[11046](info): set up 2 sockets
Sep 27 07:08:49 dist-test-slave-ltqz krb5kdc[11046](info): commencing operation
krb5kdc: starting...
W20260927 07:08:50.135519 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:50 dist-test-slave-ltqz krb5kdc[11046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:50.252260 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:41959
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:50.332345 11059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:50.332543 11059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:50.332566 11059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:50.333622 11059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:50.333662 11059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:50.333678 11059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:50.333689 11059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:50.333701 11059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:50.335058 11059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:41959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:41959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:50.335235 11059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:50.335472 11059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:50.340090 11064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:50.340126 11065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:50.340103 11067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:50.340377 11059 server_base.cc:1034] running on GCE node
I20260927 07:08:50.340648 11059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:50.340934 11059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:50.342098 11059 hybrid_clock.cc:648] HybridClock initialized: now 1790492930342067 us; error 51 us; skew 500 ppm
Sep 27 07:08:50 dist-test-slave-ltqz krb5kdc[11046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:50.344607 11059 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:50.344983 11059 webserver.cc:460] Webserver started at http://127.21.2.126:34035/ using document root <none> and password file <none>
I20260927 07:08:50.345151 11059 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:50.345191 11059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:50.345275 11059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:50.346251 11059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "bfd5535322994842899b9db379b1849d"
format_stamp: "Formatted at 2026-09-27 07:08:50 on dist-test-slave-ltqz"
server_key: "5d2cde54b0d6f8f05c21f0392d9db016"
server_key_iv: "909993c982433f319123bec043452522"
server_key_version: "encryptionkey@0"
I20260927 07:08:50.346565 11059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "bfd5535322994842899b9db379b1849d"
format_stamp: "Formatted at 2026-09-27 07:08:50 on dist-test-slave-ltqz"
server_key: "5d2cde54b0d6f8f05c21f0392d9db016"
server_key_iv: "909993c982433f319123bec043452522"
server_key_version: "encryptionkey@0"
I20260927 07:08:50.347898 11059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:50.348570 11074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:50.348749 11059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:50.348857 11059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "bfd5535322994842899b9db379b1849d"
format_stamp: "Formatted at 2026-09-27 07:08:50 on dist-test-slave-ltqz"
server_key: "5d2cde54b0d6f8f05c21f0392d9db016"
server_key_iv: "909993c982433f319123bec043452522"
server_key_version: "encryptionkey@0"
I20260927 07:08:50.348934 11059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:50.361450 11059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:50.363816 11059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:50.363976 11059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:50.367911 11059 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:41959
I20260927 07:08:50.367945 11126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:41959 every 8 connection(s)
I20260927 07:08:50.368317 11059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:50.369117 11127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:50.371317 11127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d: Bootstrap starting.
I20260927 07:08:50.371907 11127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:50.372190 11127 log.cc:824] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:50.372840 11127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d: No bootstrap required, opened a new log
I20260927 07:08:50.374392 11127 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } }
I20260927 07:08:50.374522 11127 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:50.374568 11127 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd5535322994842899b9db379b1849d, State: Initialized, Role: FOLLOWER
I20260927 07:08:50.374716 11127 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } }
I20260927 07:08:50.374797 11127 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:50.374840 11127 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:50.374888 11127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:50.375790 11127 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } }
I20260927 07:08:50.375900 11127 leader_election.cc:302] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd5535322994842899b9db379b1849d; no voters:
I20260927 07:08:50.376082 11127 leader_election.cc:288] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:50.376164 11130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:50.376338 11130 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [term 1 LEADER]: Becoming Leader. State: Replica: bfd5535322994842899b9db379b1849d, State: Running, Role: LEADER
I20260927 07:08:50.376374 11127 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:50.376480 11130 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } }
I20260927 07:08:50.376868 11131 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd5535322994842899b9db379b1849d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } } }
I20260927 07:08:50.377060 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11059
I20260927 07:08:50.377110 11131 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:50.377171 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:50.377377 21513 external_mini_cluster.cc:1428] Setting key 7706f47e9afcd2da760bda1307b79a3c
I20260927 07:08:50.377038 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd5535322994842899b9db379b1849d. Latest consensus state: current_term: 1 leader_uuid: "bfd5535322994842899b9db379b1849d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd5535322994842899b9db379b1849d" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 41959 } } }
I20260927 07:08:50.377673 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:50.378178 11139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:50.378687 11139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:50.380337 11139 catalog_manager.cc:1347] Generated new cluster ID: cef6b3bec23c4ca19a1407f34db66081
I20260927 07:08:50.380411 11139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:50.383438 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:50.378039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41953 (local address 127.21.2.126:41959)
0927 07:08:50.378261 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:50.378270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:50.378302 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:08:50.378421 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:50.378430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:50.378489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:50.378635 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:50.378852 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:50.380212 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:50.381312 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:50.381915 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:50.382012 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:50.382736 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:50.382739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:50.382740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:50.382941 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:50.382977 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:50.382990 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:50.383052 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:50.383135 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:08:50.383195 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260927 07:08:50.384441 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:50.389561 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:50.386035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59487 (local address 127.21.2.126:41959)
0927 07:08:50.386079 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:50.386081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:50.386215 (+ 134us) server_negotiation.cc:407] Connection header received
0927 07:08:50.386259 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:50.386260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:50.386272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:50.386326 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:50.386462 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:50.387198 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:50.388213 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:50.388538 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:50.388616 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:50.389312 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:50.389313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:50.389314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:50.389319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:50.389335 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:50.389340 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:50.389400 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:50.389453 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:08:50.389479 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:50.395714 11139 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:50.396222 11139 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:50.399156 11139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd5535322994842899b9db379b1849d: Generated new TSK 0
I20260927 07:08:50.399592 11139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:50.402189 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11059
2026-09-27T07:08:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:50 dist-test-slave-ltqz krb5kdc[11158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:50 dist-test-slave-ltqz krb5kdc[11158](info): set up 2 sockets
Sep 27 07:08:50 dist-test-slave-ltqz krb5kdc[11158](info): commencing operation
krb5kdc: starting...
W20260927 07:08:51.427609 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:51 dist-test-slave-ltqz krb5kdc[11158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:51.542583 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40853
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:51.623041 11171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:51.623298 11171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:51.623337 11171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:51.624547 11171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:51.624622 11171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:51.624647 11171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:51.624678 11171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:51.624709 11171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:51.626209 11171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:40853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:51.626489 11171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:51.626735 11171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:51.631551 11179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:51.631582 11177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:51.631551 11176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:51.631717 11171 server_base.cc:1034] running on GCE node
I20260927 07:08:51.631989 11171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:51.632282 11171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:51.633451 11171 hybrid_clock.cc:648] HybridClock initialized: now 1790492931633433 us; error 31 us; skew 500 ppm
Sep 27 07:08:51 dist-test-slave-ltqz krb5kdc[11158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492931, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:51.641341 11171 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:51.641795 11171 webserver.cc:460] Webserver started at http://127.21.2.126:44787/ using document root <none> and password file <none>
I20260927 07:08:51.642017 11171 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:51.642066 11171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:51.642174 11171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:51.643252 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "91710658618a44a89888a23d07207477"
format_stamp: "Formatted at 2026-09-27 07:08:51 on dist-test-slave-ltqz"
server_key: "e2c59bc6d131035510cd47134ec50584"
server_key_iv: "a24965224968724732f699e05693316c"
server_key_version: "encryptionkey@0"
I20260927 07:08:51.643697 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "91710658618a44a89888a23d07207477"
format_stamp: "Formatted at 2026-09-27 07:08:51 on dist-test-slave-ltqz"
server_key: "e2c59bc6d131035510cd47134ec50584"
server_key_iv: "a24965224968724732f699e05693316c"
server_key_version: "encryptionkey@0"
I20260927 07:08:51.645187 11171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:51.645932 11186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:51.646279 11171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:51.646411 11171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "91710658618a44a89888a23d07207477"
format_stamp: "Formatted at 2026-09-27 07:08:51 on dist-test-slave-ltqz"
server_key: "e2c59bc6d131035510cd47134ec50584"
server_key_iv: "a24965224968724732f699e05693316c"
server_key_version: "encryptionkey@0"
I20260927 07:08:51.646507 11171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:51.650427 11171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:51.652473 11171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:51.652631 11171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:51.656893 11171 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:40853
I20260927 07:08:51.656937 11238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:40853 every 8 connection(s)
I20260927 07:08:51.657316 11171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:51.658272 11239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:51.660554 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477: Bootstrap starting.
I20260927 07:08:51.661118 11239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:51.661393 11239 log.cc:824] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:51.662062 11239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477: No bootstrap required, opened a new log
I20260927 07:08:51.663741 11239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } }
I20260927 07:08:51.663872 11239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:51.663894 11239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91710658618a44a89888a23d07207477, State: Initialized, Role: FOLLOWER
I20260927 07:08:51.664062 11239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } }
I20260927 07:08:51.664142 11239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:51.664165 11239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:51.664214 11239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:51.665004 11239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } }
I20260927 07:08:51.665115 11239 leader_election.cc:302] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91710658618a44a89888a23d07207477; no voters:
I20260927 07:08:51.665293 11239 leader_election.cc:288] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:51.665364 11242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:51.665534 11239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:51.665562 11242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [term 1 LEADER]: Becoming Leader. State: Replica: 91710658618a44a89888a23d07207477, State: Running, Role: LEADER
I20260927 07:08:51.665643 11242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } }
I20260927 07:08:51.665979 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91710658618a44a89888a23d07207477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } } }
I20260927 07:08:51.665993 11244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91710658618a44a89888a23d07207477. Latest consensus state: current_term: 1 leader_uuid: "91710658618a44a89888a23d07207477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91710658618a44a89888a23d07207477" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 40853 } } }
I20260927 07:08:51.666180 11244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:51.666170 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:51.666658 11252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:51.667173 11252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:51.667447 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11171
I20260927 07:08:51.667584 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:51.667795 21513 external_mini_cluster.cc:1428] Setting key c8efb1ecfb1b297f3ae76d3964ef2fae
I20260927 07:08:51.668992 11252 catalog_manager.cc:1347] Generated new cluster ID: 03c411ad2d3341eaa4bc08c3d839238d
I20260927 07:08:51.669047 11252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:51.669932 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:51.668238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47883 (local address 127.21.2.126:40853)
0927 07:08:51.668432 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:51.668441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:51.668474 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:08:51.668605 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:51.668613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:51.668643 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:51.668805 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:51.669459 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:51.669462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:51.669462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:51.669466 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:51.669498 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:51.669513 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:51.669595 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:51.669749 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 07:08:51.669791 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260927 07:08:51.670779 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:51.677039 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:51.675758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39297 (local address 127.21.2.126:40853)
0927 07:08:51.675801 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:51.675803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:51.676011 (+ 208us) server_negotiation.cc:407] Connection header received
0927 07:08:51.676100 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:51.676102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:51.676108 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:51.676165 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:51.676801 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:51.676802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:51.676803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:08:51.676807 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:51.676822 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:08:51.676826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:51.676878 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:51.676944 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:08:51.676959 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:51.689378 11252 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:51.689577 11252 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:51.695809 11252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91710658618a44a89888a23d07207477: Generated new TSK 0
I20260927 07:08:51.696036 11252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:08:51.699196 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11171
2026-09-27T07:08:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23700 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:51 dist-test-slave-ltqz krb5kdc[11270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:51 dist-test-slave-ltqz krb5kdc[11270](info): set up 2 sockets
Sep 27 07:08:51 dist-test-slave-ltqz krb5kdc[11270](info): commencing operation
krb5kdc: starting...
W20260927 07:08:52.735769 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:52 dist-test-slave-ltqz krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:52.850569 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34095
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:52.930397 11283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:52.930673 11283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:52.930706 11283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:52.930747 11283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:52.931917 11283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:52.931989 11283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:52.932014 11283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:52.932044 11283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:52.932063 11283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:52.933564 11283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:52.933851 11283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:52.934080 11283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:52.938845 11291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:52.938871 11288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:52.938943 11283 server_base.cc:1034] running on GCE node
W20260927 07:08:52.938834 11289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:52.939297 11283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:52.939637 11283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:52.940824 11283 hybrid_clock.cc:648] HybridClock initialized: now 1790492932940808 us; error 47 us; skew 500 ppm
Sep 27 07:08:52 dist-test-slave-ltqz krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:52.943269 11283 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:52.943783 11283 webserver.cc:460] Webserver started at http://127.21.2.126:41475/ using document root <none> and password file <none>
I20260927 07:08:52.944021 11283 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:52.944092 11283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:52.944238 11283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:52.945295 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "765f1573222744a5a6f8feae0f187b76"
format_stamp: "Formatted at 2026-09-27 07:08:52 on dist-test-slave-ltqz"
server_key: "00b0edb32e78889f584f5c4fb73de20a"
server_key_iv: "e7b9281f501061a4c847b08c459826ea"
server_key_version: "encryptionkey@0"
I20260927 07:08:52.945665 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "765f1573222744a5a6f8feae0f187b76"
format_stamp: "Formatted at 2026-09-27 07:08:52 on dist-test-slave-ltqz"
server_key: "00b0edb32e78889f584f5c4fb73de20a"
server_key_iv: "e7b9281f501061a4c847b08c459826ea"
server_key_version: "encryptionkey@0"
I20260927 07:08:52.947073 11283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:08:52.947898 11298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:52.948069 11283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:52.948168 11283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "765f1573222744a5a6f8feae0f187b76"
format_stamp: "Formatted at 2026-09-27 07:08:52 on dist-test-slave-ltqz"
server_key: "00b0edb32e78889f584f5c4fb73de20a"
server_key_iv: "e7b9281f501061a4c847b08c459826ea"
server_key_version: "encryptionkey@0"
I20260927 07:08:52.948252 11283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:52.966372 11283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:52.968365 11283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:52.968532 11283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:52.972656 11283 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:34095
I20260927 07:08:52.972707 11350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:34095 every 8 connection(s)
I20260927 07:08:52.973073 11283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:52.974031 11351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:52.975564 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11283
I20260927 07:08:52.975659 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:52.975852 21513 external_mini_cluster.cc:1428] Setting key 2a9ac7990452a2b5726576659d17c820
I20260927 07:08:52.975708 11351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:52 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.125@KRBTEST.COM" created.
I20260927 07:08:52.986486 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:52.976478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34095 (local address 127.0.0.1:39628)
0927 07:08:52.976684 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:52.976722 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:52.977336 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:52.977640 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:52.977643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:52.978004 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:52.978258 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:52.978263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:52.979836 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:52.979837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:52.980501 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:52.980503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:52.981269 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:52.981270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:52.981321 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:52.981854 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:52.981862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:52.983190 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:52.985067 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:52.985073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:52.985074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:52.985258 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:52.985464 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:52.985464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:52.985464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:52.985539 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:52.985961 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:52.985966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:52.986020 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 07:08:52.986096 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:08:52.986214 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260927 07:08:52.986493 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:52.976501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.21.2.126:34095)
0927 07:08:52.976718 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:52.976729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:52.976811 (+ 82us) server_negotiation.cc:407] Connection header received
0927 07:08:52.977498 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:52.977501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:52.977519 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:52.977599 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:52.978372 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:52.979729 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:52.980592 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:52.981161 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:52.981287 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:52.983279 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:52.983289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:52.983290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:52.983298 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:52.984997 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:52.985326 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:52.985329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:52.985329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:52.985405 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:52.985637 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:52.985638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:52.985638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:52.985890 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:52.985972 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:52.986214 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 07:08:52.986291 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260927 07:08:52.987615 11351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:52.988030 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:52.987858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:57954)
0927 07:08:52.987915 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:52.987995 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 07:08:52.988199 11302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
I20260927 07:08:52.988663 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:52.988555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:57968)
0927 07:08:52.988603 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:52.988636 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:08:52.988853 11351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
I20260927 07:08:53.001966 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:46233
--webserver_interface=127.21.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:53.038132 11351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } attempt: 1
I20260927 07:08:53.038733 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.038567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:57984)
0927 07:08:53.038644 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.038695 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 07:08:53.039120 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.039032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:57996)
0927 07:08:53.039074 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.039101 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:08:53.039216 11351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
W20260927 07:08:53.082877 11358 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:53.083098 11358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:53.083124 11358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:53.083145 11358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:53.084340 11358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:53.084386 11358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:53.084400 11358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:53.084414 11358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:53.084425 11358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:53.085860 11358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:53.086062 11358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:53.086242 11358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:53.090929 11363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:53.090929 11364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:53.090924 11366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:53.091192 11358 server_base.cc:1034] running on GCE node
I20260927 07:08:53.091542 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:53.091826 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:53.092990 11358 hybrid_clock.cc:648] HybridClock initialized: now 1790492933092976 us; error 42 us; skew 500 ppm
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:53.095523 11358 init.cc:377] Logged in from keytab as kudu/127.21.2.125@KRBTEST.COM (short username kudu)
I20260927 07:08:53.095903 11358 webserver.cc:460] Webserver started at http://127.21.2.125:37163/ using document root <none> and password file <none>
I20260927 07:08:53.096079 11358 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:53.096118 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:53.096200 11358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:53.097193 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data/instance:
uuid: "555d967a69c44e5187f111a8b28318f1"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "cf71ede4333b5031f4a04277a31ec496"
server_key_iv: "b51085550d6b02516cdb0165c91085e9"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.097491 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal/instance:
uuid: "555d967a69c44e5187f111a8b28318f1"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "cf71ede4333b5031f4a04277a31ec496"
server_key_iv: "b51085550d6b02516cdb0165c91085e9"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.098685 11358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:53.099337 11373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:53.099509 11358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:53.099594 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "555d967a69c44e5187f111a8b28318f1"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "cf71ede4333b5031f4a04277a31ec496"
server_key_iv: "b51085550d6b02516cdb0165c91085e9"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.099655 11358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:53.107384 11351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } attempt: 2
I20260927 07:08:53.107817 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.107679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:57998)
0927 07:08:53.107745 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.107784 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 07:08:53.108191 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.108086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:58004)
0927 07:08:53.108139 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.108167 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 07:08:53.108319 11351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.125:46233: connect: Connection refused (error 111)
I20260927 07:08:53.108505 11358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:53.110507 11358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:53.110656 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:53.114560 11358 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.125:46233
I20260927 07:08:53.114611 11425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.125:46233 every 8 connection(s)
I20260927 07:08:53.114944 11358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
I20260927 07:08:53.116005 11426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:53.117007 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11358
I20260927 07:08:53.117117 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-1/wal/instance
I20260927 07:08:53.117295 21513 external_mini_cluster.cc:1428] Setting key e55bc7ce19117a1bde8a685d8934eebc
I20260927 07:08:53.117669 11426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.124@KRBTEST.COM" created.
I20260927 07:08:53.127849 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.118526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.21.2.126:34095)
0927 07:08:53.118579 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.118581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.118756 (+ 175us) server_negotiation.cc:407] Connection header received
0927 07:08:53.119462 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.119463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.119475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.119522 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.120215 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.121020 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.121965 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.122317 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.122417 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.124778 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.124780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.124780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.124788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.126432 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.126820 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.126821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.126821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.126874 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.127095 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.127095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.127095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.127241 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.127307 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.127644 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.127735 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:08:53.128010 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.118349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34095 (local address 127.0.0.1:39642)
0927 07:08:53.118621 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.118664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.119341 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.119600 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.119604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.119917 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.120114 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.120118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.121123 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.121124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.121876 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.121878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.122438 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.122439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.122535 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.123095 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.123111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.124670 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.126506 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.126512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.126514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.126728 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.126931 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.126931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.126931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.127037 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.127313 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.127319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.127504 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.127622 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.127785 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20260927 07:08:53.128520 11426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:53.136384 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.128773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:58006)
0927 07:08:53.128971 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.128983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.129032 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.129218 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.129219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.129376 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.129436 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.129447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.130259 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.130260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.131129 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.131132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.131662 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.131663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.131743 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.132117 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.132123 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.133440 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.135214 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.135214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.135215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.135430 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.135633 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.135634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.135634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.135687 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.136064 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.136065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.136102 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.136168 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.136271 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:08:53.136487 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.128848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.21.2.125:46233)
0927 07:08:53.129014 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.129019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.129046 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:08:53.129095 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.129098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.129122 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.129204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.129517 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.130173 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.131219 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.131585 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.131642 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.133538 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.133546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.133547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.133553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.135104 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.135503 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.135506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.135507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.135563 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.135806 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.135807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.135807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.135992 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.136085 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.136194 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.136338 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 07:08:53.137241 11426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:53.137585 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.137477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57200)
0927 07:08:53.137525 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.137567 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:08:53.137748 11376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
I20260927 07:08:53.138157 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.138079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57210)
0927 07:08:53.138119 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.138142 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 07:08:53.138309 11426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
I20260927 07:08:53.142634 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:46233
--webserver_interface=127.21.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:53.187536 11426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } attempt: 1
I20260927 07:08:53.188056 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.187923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57214)
0927 07:08:53.187987 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.188029 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:08:53.188432 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.188327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57220)
0927 07:08:53.188373 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.188398 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:08:53.188531 11426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
I20260927 07:08:53.199435 11351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } attempt: 3
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:53.207306 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.199852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:58008)
0927 07:08:53.199900 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.199913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.199997 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.200143 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.200144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.200316 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.200386 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.200387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.201073 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.201074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.202103 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.202105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.202514 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.202515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.202595 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.203081 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.203086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.204344 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.206292 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.206293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.206294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.206472 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.206706 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.206707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.206707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.206758 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.207064 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.207065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.207105 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.207157 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.207235 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 07:08:53.207455 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.199914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.21.2.125:46233)
0927 07:08:53.199979 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.199982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.199994 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:08:53.200067 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.200068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.200080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.200137 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.200481 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.200976 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.202190 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.202416 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.202513 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.204442 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.204444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.204444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.204453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.206185 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.206551 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.206552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.206552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.206630 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.206821 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.206822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.206822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.206994 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.207072 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.207200 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.207300 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:08:53.207834 11351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:53.208187 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.208080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57228)
0927 07:08:53.208130 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.208159 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:08:53.208503 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.208401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57232)
0927 07:08:53.208446 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.208476 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:08:53.208587 11351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
W20260927 07:08:53.233203 11433 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:53.233541 11433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:53.233575 11433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:53.233619 11433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:53.235652 11433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:53.235715 11433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:53.235741 11433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:53.235764 11433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:53.235782 11433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:53.238001 11433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:34095,127.21.2.125:46233,127.21.2.124:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:53.238301 11433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:53.238520 11433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:53.239706 11351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } attempt: 1
I20260927 07:08:53.240147 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.240015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57236)
0927 07:08:53.240077 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.240116 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 07:08:53.240464 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.240405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57238)
0927 07:08:53.240426 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.240447 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 07:08:53.240571 11351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
W20260927 07:08:53.243818 11438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:53.243893 11441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:53.243819 11439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:53.243958 11433 server_base.cc:1034] running on GCE node
I20260927 07:08:53.244254 11433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:53.244556 11433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:53.245775 11433 hybrid_clock.cc:648] HybridClock initialized: now 1790492933245770 us; error 47 us; skew 500 ppm
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:53.248282 11433 init.cc:377] Logged in from keytab as kudu/127.21.2.124@KRBTEST.COM (short username kudu)
I20260927 07:08:53.248746 11433 webserver.cc:460] Webserver started at http://127.21.2.124:44635/ using document root <none> and password file <none>
I20260927 07:08:53.248976 11433 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:53.249044 11433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:53.249197 11433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:53.250319 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data/instance:
uuid: "3c8a74c899f04845a1aeda1224730ad2"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "524a1a5f16d98b4b7f4b40f57dd6b872"
server_key_iv: "cd8eb6fceb352a4af26e02c8570f1c04"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.250730 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal/instance:
uuid: "3c8a74c899f04845a1aeda1224730ad2"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "524a1a5f16d98b4b7f4b40f57dd6b872"
server_key_iv: "cd8eb6fceb352a4af26e02c8570f1c04"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.252219 11433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:08:53.253005 11448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:53.253206 11433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:53.253317 11433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "3c8a74c899f04845a1aeda1224730ad2"
format_stamp: "Formatted at 2026-09-27 07:08:53 on dist-test-slave-ltqz"
server_key: "524a1a5f16d98b4b7f4b40f57dd6b872"
server_key_iv: "cd8eb6fceb352a4af26e02c8570f1c04"
server_key_version: "encryptionkey@0"
I20260927 07:08:53.253393 11433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:53.256690 11426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } attempt: 2
I20260927 07:08:53.257115 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.256975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57254)
0927 07:08:53.257034 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.257070 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 07:08:53.257494 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.257384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57268)
0927 07:08:53.257434 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.257469 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:08:53.257652 11426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:46233: Network error: Client connection negotiation failed: client connection to 127.21.2.124:46233: connect: Connection refused (error 111)
I20260927 07:08:53.269155 11433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:53.270963 11433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:53.271127 11433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:53.275332 11433 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.124:46233
I20260927 07:08:53.275406 11500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.124:46233 every 8 connection(s)
I20260927 07:08:53.275767 11433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
I20260927 07:08:53.276707 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:53.277573 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11433
I20260927 07:08:53.277694 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492641704908-21513-0/minicluster-data/master-2/wal/instance
I20260927 07:08:53.277854 21513 external_mini_cluster.cc:1428] Setting key 786030753cf3a16155616adf57fc9258
I20260927 07:08:53.278723 11501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:08:53.288702 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.279721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.21.2.126:34095)
0927 07:08:53.279797 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.279799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.279952 (+ 153us) server_negotiation.cc:407] Connection header received
0927 07:08:53.280007 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.280008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.280020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.280095 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.280520 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.281231 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.282461 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.282729 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.282808 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.285216 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.285218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.285218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.285225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.286955 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.287485 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.287486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.287486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.287549 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.288069 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.288069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.288070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.288227 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.288300 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.288512 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.288598 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:53.288887 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.278381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.21.2.124:46233)
0927 07:08:53.278567 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.278575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.278600 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:08:53.279011 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.279017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.279062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.279176 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.279975 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.281262 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.282322 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.282768 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.282867 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.285776 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.285783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.285783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.285790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.287545 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.287850 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.287852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.287852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.287932 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.288146 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.288146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.288147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.288390 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.288528 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.288610 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.288687 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260927 07:08:53.288890 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.279649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:34095 (local address 127.0.0.1:39656)
0927 07:08:53.279844 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.279887 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.279951 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.280110 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.280112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.280344 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.280430 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.280434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.281348 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.281350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.282374 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.282377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.282867 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.282868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.282932 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.283579 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.283589 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.285113 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.287045 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.287051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.287052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.287406 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.287854 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.287855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.287856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.287988 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.288298 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.288302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.288369 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.288475 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.288633 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":126,"threads_started":1}
I20260927 07:08:53.289472 11501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:53.297474 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.289714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:46233 (local address 127.0.0.1:58022)
0927 07:08:53.289749 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.289759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.289792 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.289973 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.289973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.290126 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.290192 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.290194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.291065 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.291066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.291856 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.291858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.292420 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.292421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.292488 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.292932 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.292936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.294326 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.296446 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.296447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.296447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.296608 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.296813 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.296813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.296813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.296901 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.297167 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.297168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.297202 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.297302 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.297402 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 07:08:53.297544 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.289810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.21.2.125:46233)
0927 07:08:53.289858 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.289860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.289868 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:53.289892 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.289893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.289903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.289952 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.290274 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.290986 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.291980 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.292316 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.292398 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.294422 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.294423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.294424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.294432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.296373 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.296682 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.296684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.296684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.296745 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.296965 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.296965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.296966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.297069 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.297156 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.297337 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.297432 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:53.297886 11501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:53.298254 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.289746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.21.2.126:34095)
0927 07:08:53.289788 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.289790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.289798 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:53.289815 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.289816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.289825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.289882 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.290447 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.291104 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.292084 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.292362 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.292422 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.295090 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.295092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.295092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.295100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.296789 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.297121 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.297122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.297122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.297178 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.297383 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.297384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.297384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.297508 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.297679 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.298071 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.298150 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:53.305146 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.298147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57276)
0927 07:08:53.298197 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.298207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.298243 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.298391 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.298392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.298567 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.298598 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.298599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.299356 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.299357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.300125 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.300127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.300591 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.300592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.300629 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.301017 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.301050 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.302395 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.304135 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.304136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.304136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.304296 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.304495 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.304496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.304497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.304550 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.304807 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.304808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.304846 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.304933 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.305029 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 07:08:53.305203 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.298194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.21.2.124:46233)
0927 07:08:53.298227 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.298229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.298236 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:08:53.298303 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.298304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.298313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.298359 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.298673 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.299282 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.300221 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.300516 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.300589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.302474 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.302476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.302476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.302484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.304035 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.304377 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.304378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.304379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.304430 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.304635 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.304636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.304636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.304742 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.304814 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.304955 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.305078 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":18}
I20260927 07:08:53.306463 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: Bootstrap starting.
I20260927 07:08:53.306879 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.298906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.21.2.125:46233)
0927 07:08:53.298943 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.298945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.298951 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:08:53.298972 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.298972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.298982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.299036 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.299491 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.300171 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.301076 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.301392 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.301478 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.304066 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.304068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.304068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.304076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.305706 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.306041 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.306043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.306043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.306097 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.306313 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.306314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.306314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.306425 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.306601 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.306707 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.306781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:08:53.307478 11501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:53.307785 11501 log.cc:824] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:53.308413 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:53.308653 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: No bootstrap required, opened a new log
I20260927 07:08:53.309346 11325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "765f1573222744a5a6f8feae0f187b76"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39658
I20260927 07:08:53.310214 11501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.310359 11501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8a74c899f04845a1aeda1224730ad2, State: Initialized, Role: FOLLOWER
I20260927 07:08:53.310443 11501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.310751 11501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:53.310734 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.310839 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:08:53.312520 11518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:53.312580 11518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:53.312927 11433 master_runner.cc:186] Error getting master registration for 127.21.2.126:34095: OK, instance_id { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" instance_seqno: 1790492932971454 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:08:53.314908 11433 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:08:53.315719 11351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } attempt: 2
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492932, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:08:53.323096 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.316048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.21.2.124:46233)
0927 07:08:53.316115 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.316117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.316129 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:08:53.316150 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.316151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.316166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.316239 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.316551 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.317268 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.317944 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.318156 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.318244 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.320313 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.320315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.320315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.320322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.321945 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.322280 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.322281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.322282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.322335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.322545 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.322546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.322546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.322686 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.322766 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.322921 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.322996 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:53.323097 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.315979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57280)
0927 07:08:53.316027 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.316042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.316092 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.316245 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.316246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.316413 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.316454 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.316455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.317342 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.317343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.317858 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.317859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.318243 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.318244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.318303 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.318750 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.318755 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.320244 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.322021 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.322022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.322022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.322193 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.322412 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.322413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.322413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.322471 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.322759 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.322760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.322807 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.322896 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.322996 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 07:08:53.324599 11351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: Bootstrap starting.
I20260927 07:08:53.325224 11351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:53.325511 11351 log.cc:824] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:53.326241 11351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: No bootstrap required, opened a new log
I20260927 07:08:53.327253 11351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.327411 11351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765f1573222744a5a6f8feae0f187b76, State: Initialized, Role: FOLLOWER
I20260927 07:08:53.327533 11351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.327809 11521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.327865 11351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:53.327888 11521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.328006 11325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:08:53.328125 11325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:53.328954 11325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.329828 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "765f1573222744a5a6f8feae0f187b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c8a74c899f04845a1aeda1224730ad2"
I20260927 07:08:53.329952 11475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 07:08:53.330014 11532 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:53.330060 11532 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:53.330794 11475 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 765f1573222744a5a6f8feae0f187b76 in term 1.
I20260927 07:08:53.329593 11325 leader_election.cc:288] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [CANDIDATE]: Term 1 election: Requested vote from peers 555d967a69c44e5187f111a8b28318f1 (127.21.2.125:46233), 3c8a74c899f04845a1aeda1224730ad2 (127.21.2.124:46233)
I20260927 07:08:53.331880 11301 leader_election.cc:302] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8a74c899f04845a1aeda1224730ad2, 765f1573222744a5a6f8feae0f187b76; no voters:
I20260927 07:08:53.332053 11521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:53.332201 11521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [term 1 LEADER]: Becoming Leader. State: Replica: 765f1573222744a5a6f8feae0f187b76, State: Running, Role: LEADER
I20260927 07:08:53.332278 11521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.332655 11521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 765f1573222744a5a6f8feae0f187b76. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.332716 11521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:53.333129 11283 master_runner.cc:186] Error getting master registration for 127.21.2.125:46233: OK, instance_id { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" instance_seqno: 1790492933113546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:08:53.335404 11283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:08:53.335831 11540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:53.336540 11540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:53.339006 11475 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [term 1 FOLLOWER]: Refusing update from remote peer 765f1573222744a5a6f8feae0f187b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:53.339394 11534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:53.340672 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 765f1573222744a5a6f8feae0f187b76. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.340762 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.345700 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.336201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.21.2.125:46233)
0927 07:08:53.336247 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.336249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.336900 (+ 651us) server_negotiation.cc:407] Connection header received
0927 07:08:53.336995 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.336996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.337008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.337038 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.337716 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.338475 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.339673 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.339972 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.340013 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.341718 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.341720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.341720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.341727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.343852 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.344562 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.344563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.344563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.344625 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.344955 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.344956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.344957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.345090 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.345171 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.345508 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.345600 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:53.348407 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.336090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.21.2.124:46233)
0927 07:08:53.336129 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.336130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.336736 (+ 606us) server_negotiation.cc:407] Connection header received
0927 07:08:53.336760 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.336761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.336772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.336819 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.338243 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.338902 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.341599 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.341870 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.341911 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.344307 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.344309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.344310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.344319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.345980 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.347050 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.347051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.347052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.347111 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.347310 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.347310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.347311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.347455 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.347524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.348228 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.348312 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:08:53.348762 11426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } attempt: 3
Sep 27 07:08:53 dist-test-slave-ltqz krb5kdc[11270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492933, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:08:53.354693 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.335769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.21.2.126:34095)
0927 07:08:53.335819 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.335821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.336540 (+ 719us) server_negotiation.cc:407] Connection header received
0927 07:08:53.336618 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.336620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.336631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.336688 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.339861 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.340536 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.342237 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.342523 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.342565 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.347733 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.347735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.347735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.347742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.349834 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.353782 (+ 3948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.353783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.353783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.353851 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.354076 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.354077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.354078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.354207 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.354311 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.354514 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.354596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:53.356886 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.357152 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:53.357282 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.357376 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.357334 11544 mvcc.cc:205] Tried to move back new op lower bound from 7333859054953680896 to 7333859054929743872. Current Snapshot: MvccSnapshot[applied={T|T < 7333859054953680896}]
I20260927 07:08:53.359011 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.349034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:46233 (local address 127.0.0.1:57296)
0927 07:08:53.349086 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:53.349095 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:53.349142 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:53.350073 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:53.350074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:53.350224 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:53.350268 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.350270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.351307 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.351308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.352152 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:53.352154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.352641 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.352642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:53.352697 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.353145 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:53.353151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:53.354599 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:53.357790 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.357791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.357792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.358022 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.358305 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:53.358306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:53.358307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:53.358382 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:53.358688 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:53.358689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:53.358735 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 07:08:53.358797 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 07:08:53.358896 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:08:53.359279 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:53.349168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.21.2.124:46233)
0927 07:08:53.349931 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.349932 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.349939 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:08:53.349958 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.349959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.349968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.350027 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.350370 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.351211 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.352251 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:53.352540 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:53.352595 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:53.354829 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:53.354830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:53.354831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:53.354837 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:53.357716 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.358136 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.358138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.358138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.358210 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:53.358466 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:53.358467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:53.358468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:53.358602 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:53.358662 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:53.359125 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 07:08:53.359195 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743}
I20260927 07:08:53.360843 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: Bootstrap starting.
I20260927 07:08:53.361557 11426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:53.361853 11426 log.cc:824] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:53.362768 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: No bootstrap required, opened a new log
I20260927 07:08:53.364785 11426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.364907 11426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555d967a69c44e5187f111a8b28318f1, State: Initialized, Role: FOLLOWER
I20260927 07:08:53.364976 11426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } }
I20260927 07:08:53.365283 11426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:53.365350 11399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:53.366214 11399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [term 1 FOLLOWER]: Refusing update from remote peer 765f1573222744a5a6f8feae0f187b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:53.366375 11550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.366451 11550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.366518 11534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:53.366838 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "765f1573222744a5a6f8feae0f187b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "555d967a69c44e5187f111a8b28318f1"
I20260927 07:08:53.367132 11540 catalog_manager.cc:1347] Generated new cluster ID: 88ddbcdd12c7473e93fa81895c254aa8
I20260927 07:08:53.367177 11540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:53.367853 11550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 765f1573222744a5a6f8feae0f187b76. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.367933 11550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.367966 11550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.368017 11550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:53.369244 11559 mvcc.cc:205] Tried to move back new op lower bound from 7333859054953680896 to 7333859054929743872. Current Snapshot: MvccSnapshot[applied={T|T < 7333859054953680896}]
I20260927 07:08:53.369634 11400 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 765f1573222744a5a6f8feae0f187b76 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 07:08:53.378499 11534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "765f1573222744a5a6f8feae0f187b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765f1573222744a5a6f8feae0f187b76" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 34095 } } peers { permanent_uuid: "555d967a69c44e5187f111a8b28318f1" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 46233 } } peers { permanent_uuid: "3c8a74c899f04845a1aeda1224730ad2" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 46233 } } }
I20260927 07:08:53.378602 11534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:53.379328 11540 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:53.380031 11540 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:53.383634 11565 catalog_manager.cc:1259] Loaded cluster ID: 88ddbcdd12c7473e93fa81895c254aa8
I20260927 07:08:53.383754 11565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: loading cluster ID for follower catalog manager: success
W20260927 07:08:53.383926 11565 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260927 07:08:53.384019 11565 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:53.387423 11540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 765f1573222744a5a6f8feae0f187b76: Generated new TSK 0
I20260927 07:08:53.387585 11540 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:08:53.416879 11353 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.415926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.21.2.126:34095)
0927 07:08:53.415975 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.415977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.416171 (+ 194us) server_negotiation.cc:407] Connection header received
0927 07:08:53.416290 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.416292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.416303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.416334 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.416807 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39690: BlockingRecv error: recv got EOF from 127.0.0.1:39690 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 07:08:53.418164 11504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.417270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.21.2.124:46233)
0927 07:08:53.417310 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.417312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.417644 (+ 332us) server_negotiation.cc:407] Connection header received
0927 07:08:53.417740 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.417742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.417752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.417781 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.418090 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:08:53.418946 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.417477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.21.2.125:46233)
0927 07:08:53.417516 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.417517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.418413 (+ 896us) server_negotiation.cc:407] Connection header received
0927 07:08:53.418511 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.418512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.418523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.418552 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.418871 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58042: BlockingRecv error: recv got EOF from 127.0.0.1:58042 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 07:08:53.421330 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.420574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.21.2.125:46233)
0927 07:08:53.420611 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.420613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.420806 (+ 193us) server_negotiation.cc:407] Connection header received
0927 07:08:53.420913 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.420914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.420931 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.420961 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.421286 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58050: BlockingRecv error: recv got EOF from 127.0.0.1:58050 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 07:08:53.423640 11425 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.21.2.125:46233: Invalid argument: socket address is ambiguous
W20260927 07:08:53.424546 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.423772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.21.2.125:46233)
0927 07:08:53.423805 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.423807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.424044 (+ 237us) server_negotiation.cc:407] Connection header received
0927 07:08:53.424163 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.424164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.424174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.424202 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.424506 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58054: BlockingRecv error: recv got EOF from 127.0.0.1:58054 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 07:08:53.428824 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.428005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.21.2.125:46233)
0927 07:08:53.428046 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.428048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.428316 (+ 268us) server_negotiation.cc:407] Connection header received
0927 07:08:53.428413 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.428414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.428424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.428453 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.428780 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58068: BlockingRecv error: recv got EOF from 127.0.0.1:58068 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 07:08:53.434865 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.434049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.21.2.125:46233)
0927 07:08:53.434087 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.434089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.434334 (+ 245us) server_negotiation.cc:407] Connection header received
0927 07:08:53.434432 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.434433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.434443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.434472 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.434815 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58072: BlockingRecv error: recv got EOF from 127.0.0.1:58072 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:08:53.445019 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.444145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.21.2.125:46233)
0927 07:08:53.444185 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.444187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.444457 (+ 270us) server_negotiation.cc:407] Connection header received
0927 07:08:53.444561 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.444562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.444574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.444605 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.444963 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58076: BlockingRecv error: recv got EOF from 127.0.0.1:58076 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 07:08:53.463351 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.462389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.21.2.125:46233)
0927 07:08:53.462435 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.462437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.462737 (+ 300us) server_negotiation.cc:407] Connection header received
0927 07:08:53.462854 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.462856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.462868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.462898 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.463290 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58082: BlockingRecv error: recv got EOF from 127.0.0.1:58082 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 07:08:53.497390 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.496657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.21.2.125:46233)
0927 07:08:53.496704 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.496707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.496839 (+ 132us) server_negotiation.cc:407] Connection header received
0927 07:08:53.496916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.496917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.496930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.497002 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.497327 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58086: BlockingRecv error: recv got EOF from 127.0.0.1:58086 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:08:53.563462 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.562706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.21.2.125:46233)
0927 07:08:53.562793 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.562795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.562861 (+ 66us) server_negotiation.cc:407] Connection header received
0927 07:08:53.562904 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.562905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.562915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.562982 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.563360 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58094: BlockingRecv error: recv got EOF from 127.0.0.1:58094 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 07:08:53.693285 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.692555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.21.2.125:46233)
0927 07:08:53.692654 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.692656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.692675 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:08:53.692752 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.692753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.692764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.692818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.693217 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58102: BlockingRecv error: recv got EOF from 127.0.0.1:58102 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 07:08:53.951155 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:53.950462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.21.2.125:46233)
0927 07:08:53.950533 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:53.950537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:53.950552 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:08:53.950578 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:53.950578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:53.950588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:53.950694 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:53.951076 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58116: BlockingRecv error: recv got EOF from 127.0.0.1:58116 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 07:08:54.116503 11421 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.2.125:46233
W20260927 07:08:54.277086 11496 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.21.2.124:46233
I20260927 07:08:54.313069 11518 catalog_manager.cc:1259] Loaded cluster ID: 88ddbcdd12c7473e93fa81895c254aa8
I20260927 07:08:54.313180 11518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: loading cluster ID for follower catalog manager: success
I20260927 07:08:54.313994 11518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: acquiring CA information for follower catalog manager: success
I20260927 07:08:54.314301 11518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c8a74c899f04845a1aeda1224730ad2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:08:54.385174 11565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: acquiring CA information for follower catalog manager: success
I20260927 07:08:54.385558 11565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 555d967a69c44e5187f111a8b28318f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:08:54.467283 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:54.464387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.21.2.125:46233)
0927 07:08:54.464508 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:54.464511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:54.464587 (+ 76us) server_negotiation.cc:407] Connection header received
0927 07:08:54.464687 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:54.464688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:54.464739 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:54.464830 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:54.465000 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.465591 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.466449 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.466650 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.466729 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:54.466916 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:54.467023 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:54.467105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:54.467162 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 07:08:54.467183 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260927 07:08:54.471236 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:54.469032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.21.2.124:46233)
0927 07:08:54.469133 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:54.469135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:54.469143 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:54.469177 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:54.469177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:54.469210 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:54.469276 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:54.469375 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.469838 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.470516 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.470707 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.470758 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:54.470889 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:54.470980 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:54.471034 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:54.471082 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 07:08:54.471099 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
W20260927 07:08:54.472888 11500 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.21.2.124:46233: Invalid argument: socket address is ambiguous
I20260927 07:08:54.475904 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:54.472876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.21.2.125:46233)
0927 07:08:54.472926 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:54.472929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:54.473149 (+ 220us) server_negotiation.cc:407] Connection header received
0927 07:08:54.473252 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:54.473253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:54.473265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:54.473296 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:54.473521 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.474014 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475113 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.475343 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475424 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:54.475608 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:54.475641 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:54.475693 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:54.475798 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:08:54.475822 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:54.476001 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:54.473025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.21.2.124:46233)
0927 07:08:54.473127 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:54.473129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:54.473222 (+ 93us) server_negotiation.cc:407] Connection header received
0927 07:08:54.473258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:54.473259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:54.473269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:54.473319 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:54.473475 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.474188 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475141 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.475446 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475499 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:54.475661 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:54.475689 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:54.475737 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:54.475801 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 07:08:54.475824 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:08:54.476092 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:54.472932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.21.2.126:34095)
0927 07:08:54.473036 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:54.473038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:54.473209 (+ 171us) server_negotiation.cc:407] Connection header received
0927 07:08:54.473258 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:54.473259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:54.473336 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:54.473420 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:54.473586 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.474303 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475070 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:54.475407 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:54.475468 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:54.475700 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:54.475809 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:54.475862 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:54.475959 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:08:54.475987 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260927 07:08:54.477843 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11283
I20260927 07:08:54.483083 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11358
I20260927 07:08:54.488034 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11433
2026-09-27T07:08:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2790 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:54 dist-test-slave-ltqz krb5kdc[11647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:54 dist-test-slave-ltqz krb5kdc[11647](info): set up 2 sockets
Sep 27 07:08:54 dist-test-slave-ltqz krb5kdc[11647](info): commencing operation
krb5kdc: starting...
W20260927 07:08:55.523710 21513 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/test-user.keytab.
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
Entry for principal kudu/127.21.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab.
I20260927 07:08:55.639518 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:39939
--webserver_interface=127.21.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:55.720783 11660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:55.721078 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:55.721112 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:55.721153 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:55.722322 11660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:55.722391 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:55.722414 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:55.722435 11660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:55.722465 11660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:55.724035 11660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.126.keytab
--principal=kudu/127.21.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.126:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:55.724320 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:55.724550 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:55.729282 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:55.729305 11660 server_base.cc:1034] running on GCE node
W20260927 07:08:55.729259 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:55.729259 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:55.729854 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:55.730135 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:55.731277 11660 hybrid_clock.cc:648] HybridClock initialized: now 1790492935731258 us; error 41 us; skew 500 ppm
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:55.733826 11660 init.cc:377] Logged in from keytab as kudu/127.21.2.126@KRBTEST.COM (short username kudu)
I20260927 07:08:55.734293 11660 webserver.cc:460] Webserver started at http://127.21.2.126:44957/ using document root <none> and password file <none>
I20260927 07:08:55.734519 11660 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:55.734565 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:55.734709 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:55.735824 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data/instance:
uuid: "c830472b9c7f4fdc897f345209a1a3b2"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "2dd459a52ad5ea2201ef28db8e0d25ab"
server_key_iv: "26ed531143218394f520e8eb96ce6ab9"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.736192 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance:
uuid: "c830472b9c7f4fdc897f345209a1a3b2"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "2dd459a52ad5ea2201ef28db8e0d25ab"
server_key_iv: "26ed531143218394f520e8eb96ce6ab9"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.737527 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:55.738301 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:55.738500 11660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:08:55.738587 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal
uuid: "c830472b9c7f4fdc897f345209a1a3b2"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "2dd459a52ad5ea2201ef28db8e0d25ab"
server_key_iv: "26ed531143218394f520e8eb96ce6ab9"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.738660 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:55.747246 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:55.749502 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:55.749653 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:55.753690 11660 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.126:39939
I20260927 07:08:55.753755 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.126:39939 every 8 connection(s)
I20260927 07:08:55.754087 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:55.755035 11728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:55.756682 11728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:55.764227 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11660
I20260927 07:08:55.764329 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-0/wal/instance
I20260927 07:08:55.764533 21513 external_mini_cluster.cc:1428] Setting key 07fe738f00ffc0082bc502f1a4270f81
I20260927 07:08:55.766471 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.757282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:39939 (local address 127.0.0.1:50224)
0927 07:08:55.757472 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.757514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:55.758092 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:55.758401 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:55.758404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:55.758644 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:55.758791 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.758795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.759871 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.759872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.760537 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.760539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.761065 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.761066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.761126 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.761655 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:55.761664 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:55.763053 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:55.764918 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.764926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.764928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.765206 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.765428 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.765429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.765430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.765548 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.765885 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:55.765890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:55.765952 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 07:08:55.766052 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:08:55.766286 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":93,"thread_start_us":107,"threads_started":1}
I20260927 07:08:55.766474 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.757359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.21.2.126:39939)
0927 07:08:55.757512 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:55.757523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:55.757594 (+ 71us) server_negotiation.cc:407] Connection header received
0927 07:08:55.758242 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:55.758244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:55.758261 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:55.758355 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:55.758911 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.759778 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.760635 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.760948 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.761032 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.763139 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:55.763153 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:55.763153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:55.763159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:55.764843 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.765282 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.765284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.765284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.765367 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.765612 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.765613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.765613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.765811 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:55.765904 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:55.766096 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 07:08:55.766223 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260927 07:08:55.767207 11728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:55.767692 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.767564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44622)
0927 07:08:55.767604 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.767660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 07:08:55.767809 11678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
I20260927 07:08:55.768651 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.768558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44628)
0927 07:08:55.768602 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.768626 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:08:55.768755 11728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:42841: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
Entry for principal kudu/127.21.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab.
I20260927 07:08:55.788076 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:42841
--webserver_interface=127.21.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:55.817988 11728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } attempt: 1
I20260927 07:08:55.818463 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.818332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44634)
0927 07:08:55.818397 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.818435 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:08:55.818816 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.818721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44636)
0927 07:08:55.818767 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.818797 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:08:55.818909 11728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:42841: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
W20260927 07:08:55.867110 11735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:55.867306 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:55.867323 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:55.867343 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:55.868474 11735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:55.868515 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:55.868528 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:55.868541 11735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:55.868590 11735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:55.870049 11735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.125.keytab
--principal=kudu/127.21.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.125:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:55.870285 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:55.870512 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:55.875224 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:55.875241 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:55.875320 11735 server_base.cc:1034] running on GCE node
W20260927 07:08:55.875238 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:55.875708 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:55.875983 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:55.877135 11735 hybrid_clock.cc:648] HybridClock initialized: now 1790492935877123 us; error 29 us; skew 500 ppm
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:55.879793 11735 init.cc:377] Logged in from keytab as kudu/127.21.2.125@KRBTEST.COM (short username kudu)
I20260927 07:08:55.880209 11735 webserver.cc:460] Webserver started at http://127.21.2.125:41111/ using document root <none> and password file <none>
I20260927 07:08:55.880434 11735 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:55.880479 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:55.880575 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:55.881664 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data/instance:
uuid: "4151140fb752426ebf761e7fcda11dd6"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "09fe24f9c79cefbac35616507a799975"
server_key_iv: "43b391df555c7c1139ba58f998eb3c7b"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.882061 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal/instance:
uuid: "4151140fb752426ebf761e7fcda11dd6"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "09fe24f9c79cefbac35616507a799975"
server_key_iv: "43b391df555c7c1139ba58f998eb3c7b"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.883518 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:08:55.884305 11750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:55.884487 11735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:08:55.884570 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal
uuid: "4151140fb752426ebf761e7fcda11dd6"
format_stamp: "Formatted at 2026-09-27 07:08:55 on dist-test-slave-ltqz"
server_key: "09fe24f9c79cefbac35616507a799975"
server_key_iv: "43b391df555c7c1139ba58f998eb3c7b"
server_key_version: "encryptionkey@0"
I20260927 07:08:55.884624 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:55.887054 11728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } attempt: 2
I20260927 07:08:55.887527 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.887398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44644)
0927 07:08:55.887449 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.887492 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 07:08:55.887882 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.887763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44646)
0927 07:08:55.887818 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.887850 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 07:08:55.887987 11728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.125:42841: Network error: Client connection negotiation failed: client connection to 127.21.2.125:42841: connect: Connection refused (error 111)
I20260927 07:08:55.894562 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:55.896512 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:55.896620 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:55.900372 11735 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.125:42841
I20260927 07:08:55.900419 11802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.125:42841 every 8 connection(s)
I20260927 07:08:55.900775 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/data/info.pb
I20260927 07:08:55.901732 11803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:55.902321 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11735
I20260927 07:08:55.902421 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-1/wal/instance
I20260927 07:08:55.902631 21513 external_mini_cluster.cc:1428] Setting key 23d40ed3edb6c590e97c3c7a5053b35f
I20260927 07:08:55.903440 11803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.2.124@KRBTEST.COM" created.
I20260927 07:08:55.913098 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.904134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.21.2.126:39939)
0927 07:08:55.904203 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:55.904205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:55.904368 (+ 163us) server_negotiation.cc:407] Connection header received
0927 07:08:55.904831 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:55.904833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:55.904852 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:55.904913 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:55.905498 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.906310 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.907293 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.907697 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.907781 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.910075 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:55.910077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:55.910077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:55.910085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:55.911545 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.911990 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.911991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.911992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.912051 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.912369 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.912370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.912371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.912490 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:55.912557 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:55.912908 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 07:08:55.912994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:08:55.913301 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.904073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:39939 (local address 127.0.0.1:50230)
0927 07:08:55.904247 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.904285 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:55.904733 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:55.904985 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:55.904988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:55.905217 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:55.905403 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.905406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.906418 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.906419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.907181 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.907184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.907800 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.907801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.907901 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.908463 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:55.908471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:55.909985 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:55.911616 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.911622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.911624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.911854 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.912144 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.912145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.912145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.912277 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.912598 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:55.912606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:55.912744 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:08:55.912868 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 07:08:55.913065 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260927 07:08:55.913856 11803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:55.923066 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.914132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44656)
0927 07:08:55.914180 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.914192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:55.914236 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:55.915670 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:55.915671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:55.915782 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:55.915830 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.915831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.917137 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.917138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.917621 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.917622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.918316 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.918317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.918373 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.918771 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:55.918776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:55.920225 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:55.922017 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.922018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.922019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.922176 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.922433 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.922434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.922434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.922520 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.922821 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:55.922822 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:55.922860 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:08:55.922913 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:08:55.922997 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 07:08:55.923240 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.914203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.21.2.125:42841)
0927 07:08:55.915504 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:55.915509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:55.915534 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:08:55.915557 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:55.915558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:55.915572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:55.915645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:55.915900 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.917050 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.917697 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.918225 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.918301 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.920305 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:55.920313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:55.920313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:55.920320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:55.921902 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.922288 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.922290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.922291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.922349 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.922607 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.922608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.922608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.922754 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:55.922828 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:55.922945 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 07:08:55.923048 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":1260,"thread_start_us":77,"threads_started":1}
I20260927 07:08:55.923897 11803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:55.924249 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.924147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58682)
0927 07:08:55.924180 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.924228 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 07:08:55.924387 11751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
I20260927 07:08:55.924821 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.924736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58686)
0927 07:08:55.924778 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.924804 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:08:55.924930 11803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:45981: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
Entry for principal kudu/127.21.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab.
I20260927 07:08:55.930137 21513 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHv0k7x/build/release/bin/kudu
/tmp/dist-test-taskHv0k7x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:45981
--webserver_interface=127.21.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.2.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:55.974221 11803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } attempt: 1
I20260927 07:08:55.974741 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.974606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58698)
0927 07:08:55.974679 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.974712 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 07:08:55.975080 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.974987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58710)
0927 07:08:55.975031 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.975055 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:08:55.975167 11803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:45981: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
I20260927 07:08:55.979142 11728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } attempt: 3
Sep 27 07:08:55 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:55.986004 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.979569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.21.2.125:42841)
0927 07:08:55.979619 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:55.979621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:55.979637 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:08:55.979668 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:55.979669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:55.979681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:55.979746 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:55.980082 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.980582 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.981285 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.981517 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.981593 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.983413 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:55.983415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:55.983415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:55.983427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:55.984828 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.985252 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.985254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.985254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.985310 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:55.985506 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:55.985507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:55.985507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:55.985617 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:55.985691 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:55.985837 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 07:08:55.985913 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:08:55.986016 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.979492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44670)
0927 07:08:55.979542 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.979553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:55.979603 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:55.979752 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:55.979753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:55.979929 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:55.979994 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.979996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.980671 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.980673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.981203 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:55.981205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:55.981588 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:55.981590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:55.981667 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:55.982162 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:55.982166 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:55.983301 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:55.984958 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.984959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.984960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.985165 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.985373 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:55.985374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:55.985374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:55.985438 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:55.985681 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:55.985682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:55.985727 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:08:55.985813 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 07:08:55.985913 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 07:08:55.986353 11728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:55.986689 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.986594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58722)
0927 07:08:55.986639 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.986665 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 07:08:55.987085 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:55.987003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58738)
0927 07:08:55.987046 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:55.987069 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:08:55.987179 11728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:45981: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
W20260927 07:08:56.011350 11810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:08:56.011633 11810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:56.011649 11810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:56.011693 11810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:56.012837 11810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:56.012883 11810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:56.012897 11810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:56.012909 11810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:56.012921 11810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:56.014447 11810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.2.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.2.126:39939,127.21.2.125:42841,127.21.2.124:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/krb5kdc/kudu_127.21.2.124.keytab
--principal=kudu/127.21.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.2.124:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:08:56.014688 11810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:56.014914 11810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:08:56.018363 11728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } attempt: 1
I20260927 07:08:56.018951 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.018822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58742)
0927 07:08:56.018886 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.018921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:08:56.019536 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.019400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58756)
0927 07:08:56.019474 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.019503 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:08:56.019658 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:56.019672 11816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:56.019912 11728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.2.124:45981: Network error: Client connection negotiation failed: client connection to 127.21.2.124:45981: connect: Connection refused (error 111)
W20260927 07:08:56.019913 11818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:56.019902 11810 server_base.cc:1034] running on GCE node
I20260927 07:08:56.020140 11810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:56.020421 11810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:56.021591 11810 hybrid_clock.cc:648] HybridClock initialized: now 1790492936021590 us; error 40 us; skew 500 ppm
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492936, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:56.024005 11810 init.cc:377] Logged in from keytab as kudu/127.21.2.124@KRBTEST.COM (short username kudu)
I20260927 07:08:56.024487 11810 webserver.cc:460] Webserver started at http://127.21.2.124:35853/ using document root <none> and password file <none>
I20260927 07:08:56.024703 11810 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:56.024763 11810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:56.024909 11810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:08:56.025933 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data/instance:
uuid: "6d9838bfb87b4ff081c831f9b85955eb"
format_stamp: "Formatted at 2026-09-27 07:08:56 on dist-test-slave-ltqz"
server_key: "756e20707181004ed5a841811a1651b5"
server_key_iv: "295510118ad3b6fb35f87274ed3d9e6b"
server_key_version: "encryptionkey@0"
I20260927 07:08:56.026288 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal/instance:
uuid: "6d9838bfb87b4ff081c831f9b85955eb"
format_stamp: "Formatted at 2026-09-27 07:08:56 on dist-test-slave-ltqz"
server_key: "756e20707181004ed5a841811a1651b5"
server_key_iv: "295510118ad3b6fb35f87274ed3d9e6b"
server_key_version: "encryptionkey@0"
I20260927 07:08:56.027659 11810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:08:56.028335 11825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:56.028489 11810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:08:56.028585 11810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data,/tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal
uuid: "6d9838bfb87b4ff081c831f9b85955eb"
format_stamp: "Formatted at 2026-09-27 07:08:56 on dist-test-slave-ltqz"
server_key: "756e20707181004ed5a841811a1651b5"
server_key_iv: "295510118ad3b6fb35f87274ed3d9e6b"
server_key_version: "encryptionkey@0"
I20260927 07:08:56.028653 11810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:56.035507 11810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:56.037210 11810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:56.037358 11810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:56.041350 11810 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.124:45981
I20260927 07:08:56.041427 11877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.2.124:45981 every 8 connection(s)
I20260927 07:08:56.041764 11810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/data/info.pb
I20260927 07:08:56.042776 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:56.043308 11803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } attempt: 2
I20260927 07:08:56.044544 11878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:56.044937 21513 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHv0k7x/build/release/bin/kudu as pid 11810
I20260927 07:08:56.045047 21513 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHv0k7x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492641704908-21513-0/minicluster-data/master-2/wal/instance
I20260927 07:08:56.045275 21513 external_mini_cluster.cc:1428] Setting key 5f440a5a5bab2a64ff826bab303c7b9f
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.125@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492936, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:08:56.054194 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.045374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.21.2.126:39939)
0927 07:08:56.045420 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.045422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.045448 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:08:56.045526 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.045528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.045539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.045593 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.046253 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.046981 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.047854 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.048148 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.048213 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.050544 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.050545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.050546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.050554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.052523 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053141 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.053142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.053143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.053205 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053497 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.053499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.053499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.053656 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.053701 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.053993 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.054081 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:56.054603 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.045583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.21.2.124:45981)
0927 07:08:56.045771 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.045772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.045840 (+ 68us) server_negotiation.cc:407] Connection header received
0927 07:08:56.045887 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.045887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.045897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.045977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.046671 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.047231 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.048247 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.048585 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.048654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.051046 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.051047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.051047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.051055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.052886 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053306 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.053308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.053309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.053388 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053606 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.053607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.053607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.053808 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.053947 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.054044 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.054168 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260927 07:08:56.054608 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.045145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.126:39939 (local address 127.0.0.1:50244)
0927 07:08:56.045297 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.045389 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:56.045461 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:56.045679 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:56.045682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:56.046082 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:56.046167 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.046171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.047074 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.047075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.047767 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.047769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.048241 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.048242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.048326 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.048972 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:56.048982 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:56.050457 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:56.052627 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.052635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.052637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.053051 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.053274 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.053275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.053392 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.053751 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:56.053758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:56.053826 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 07:08:56.054324 (+ 498us) client_negotiation.cc:240] Negotiation successful
0927 07:08:56.054442 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260927 07:08:56.054780 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.043668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58770)
0927 07:08:56.043726 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.043737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:56.043785 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:56.044591 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:56.044593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:56.044766 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:56.044809 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.044811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.046392 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.046393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.047151 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.047153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.047947 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.047948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.048016 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.048472 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:56.048478 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:56.050861 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:56.053698 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.053699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.053699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.053925 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.054135 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.054136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.054136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.054193 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.054474 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:56.054474 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:56.054526 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 07:08:56.054573 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:08:56.054676 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:08:56.055152 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.043806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.21.2.124:45981)
0927 07:08:56.043993 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.044001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.044025 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:08:56.044409 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.044415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.044459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.044601 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.044905 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.046294 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.047228 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.047822 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.047960 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.050990 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.050997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.050997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.051004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.053606 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.054008 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.054008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.054009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.054064 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.054265 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.054265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.054266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.054376 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.054898 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.054943 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.055016 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260927 07:08:56.055223 11878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:56.056946 11803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: Bootstrap starting.
I20260927 07:08:56.057900 11803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:56.058223 11803 log.cc:824] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:56.059026 11803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: No bootstrap required, opened a new log
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492936, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.126@KRBTEST.COM
I20260927 07:08:56.060863 11803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.060997 11803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4151140fb752426ebf761e7fcda11dd6, State: Initialized, Role: FOLLOWER
I20260927 07:08:56.061081 11803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.061386 11803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:56.061353 11885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.061478 11885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:08:56.063306 11896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:56.063395 11896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:56.063706 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.055475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.125:42841 (local address 127.0.0.1:44680)
0927 07:08:56.055556 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.055565 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:56.055604 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:56.055748 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:56.055749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:56.055892 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:56.055935 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.055937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.056787 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.056788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.057558 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.057561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.058071 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.058072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.058116 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.058643 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:56.058649 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:56.060422 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:56.062460 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.062461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.062461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.062677 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.062900 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.062901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.062901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.062964 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.063278 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:56.063278 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:56.063320 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:08:56.063505 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 07:08:56.063619 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 07:08:56.064177 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.055557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.21.2.125:42841)
0927 07:08:56.055594 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.055596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.055604 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:56.055676 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.055678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.055687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.055738 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.056027 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.056686 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.057645 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.057953 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.058018 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.060545 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.060546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.060547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.060554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.062363 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.062742 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.062743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.062744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.062808 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.063033 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.063034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.063035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.063185 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.063248 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.064007 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.064092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:56.064285 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.055502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.21.2.126:39939)
0927 07:08:56.055537 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.055539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.055721 (+ 182us) server_negotiation.cc:407] Connection header received
0927 07:08:56.055789 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.055790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.055801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.055857 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.056297 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.056874 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.057573 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.057847 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.057910 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.060580 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.060581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.060581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.060588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.062719 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.063097 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.063099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.063099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.063158 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.063394 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.063395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.063395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.063545 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.063675 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.064114 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.064191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:08:56.064533 11878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:08:56.063884 11735 master_runner.cc:186] Error getting master registration for 127.21.2.126:39939: OK, instance_id { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" instance_seqno: 1790492935752546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:08:56.066591 11735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492936, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.124@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.2.125@KRBTEST.COM
I20260927 07:08:56.072544 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.064779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58786)
0927 07:08:56.064862 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.064873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:56.064920 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:56.065044 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:56.065044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:56.065201 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:56.065233 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.065234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.066088 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.066089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.066807 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.066826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.067445 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.067445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.067494 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.067925 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:56.067930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:56.069598 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:56.071472 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.071473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.071473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.071674 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.071881 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.071881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.071882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.071946 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.072252 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:56.072253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:56.072301 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 07:08:56.072377 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:08:56.072482 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 07:08:56.072641 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.064826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.21.2.124:45981)
0927 07:08:56.064862 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.064864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.064906 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:08:56.064970 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.064971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.064980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.065039 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.065307 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.065999 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.066994 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.067301 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.067390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.069685 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.069686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.069686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.069694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.071315 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.071744 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.071746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.071746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.071805 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.072021 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.072022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.072022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.072190 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.072258 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.072415 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.072548 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":18}
I20260927 07:08:56.074077 11878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: Bootstrap starting.
I20260927 07:08:56.074314 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.067136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.21.2.125:42841)
0927 07:08:56.067170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.067172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.067180 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:56.067241 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.067241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.067249 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.067293 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.067775 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.068209 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.068828 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.069103 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.069164 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.071318 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.071319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.071319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.071326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.073266 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.073603 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.073604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.073604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.073655 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.073845 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.073845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.073846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.073952 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.074094 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.074138 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.074215 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:08:56.074878 11878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:56.075183 11878 log.cc:824] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:56.075707 21513 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:08:56.075991 11878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: No bootstrap required, opened a new log
I20260927 07:08:56.077458 11878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.076661 11702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c830472b9c7f4fdc897f345209a1a3b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50250
I20260927 07:08:56.077593 11878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9838bfb87b4ff081c831f9b85955eb, State: Initialized, Role: FOLLOWER
I20260927 07:08:56.077682 11878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.077906 11899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.077986 11878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:56.077993 11899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:08:56.080016 11910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:08:56.080067 11910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:56.079994 11810 master_runner.cc:186] Error getting master registration for 127.21.2.126:39939: OK, instance_id { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" instance_seqno: 1790492935752546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:08:56.081552 11810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:08:56.095080 11728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } attempt: 2
Sep 27 07:08:56 dist-test-slave-ltqz krb5kdc[11647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492935, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.2.126@KRBTEST.COM for kudu/127.21.2.124@KRBTEST.COM
I20260927 07:08:56.103451 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.095640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.21.2.124:45981)
0927 07:08:56.095677 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.095679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.095687 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:56.095711 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.095712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.095721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.095767 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.096143 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.096908 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.097946 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.098269 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.098339 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.100490 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.100492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.100492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.100500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.102194 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.102617 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.102618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.102618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.102676 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.102876 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.102877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.102877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.103017 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.103084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.103249 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.103326 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:08:56.103464 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.095397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.2.124:45981 (local address 127.0.0.1:58802)
0927 07:08:56.095513 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:56.095524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:56.095567 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:56.095822 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:56.095823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:56.096007 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:56.096058 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.096061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.097010 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.097011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.097826 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:56.097829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.098340 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.098341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:56.098399 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.098872 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:56.098877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:56.100396 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:56.102285 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.102287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.102287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.102529 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.102743 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:56.102744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:56.102745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:56.102812 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:56.103084 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:56.103085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:56.103137 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 07:08:56.103213 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:08:56.103313 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 07:08:56.105418 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: Bootstrap starting.
I20260927 07:08:56.106153 11728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:56.106462 11728 log.cc:824] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:56.107327 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: No bootstrap required, opened a new log
I20260927 07:08:56.108716 11728 raft_consensus.cc:348] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.108863 11728 raft_consensus.cc:729] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c830472b9c7f4fdc897f345209a1a3b2, State: Initialized, Role: FOLLOWER
I20260927 07:08:56.108947 11728 consensus_queue.cc:260] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.109275 11728 sys_catalog.cc:564] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:56.109238 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.109429 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:56.110617 11702 raft_consensus.cc:482] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:08:56.110687 11702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 07:08:56.111413 11924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 07:08:56.111491 11702 raft_consensus.cc:504] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.111749 11702 leader_election.cc:288] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 4151140fb752426ebf761e7fcda11dd6 (127.21.2.125:42841), 6d9838bfb87b4ff081c831f9b85955eb (127.21.2.124:45981)
I20260927 07:08:56.112170 11852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c830472b9c7f4fdc897f345209a1a3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d9838bfb87b4ff081c831f9b85955eb"
I20260927 07:08:56.112198 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c830472b9c7f4fdc897f345209a1a3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4151140fb752426ebf761e7fcda11dd6"
I20260927 07:08:56.112313 11777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:56.112318 11852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:56.113175 11777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c830472b9c7f4fdc897f345209a1a3b2 in term 1.
I20260927 07:08:56.113593 11852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c830472b9c7f4fdc897f345209a1a3b2 in term 1.
I20260927 07:08:56.113849 11676 leader_election.cc:302] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d9838bfb87b4ff081c831f9b85955eb, c830472b9c7f4fdc897f345209a1a3b2; no voters:
I20260927 07:08:56.113973 11913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:56.114089 11913 raft_consensus.cc:686] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [term 1 LEADER]: Becoming Leader. State: Replica: c830472b9c7f4fdc897f345209a1a3b2, State: Running, Role: LEADER
I20260927 07:08:56.114171 11913 consensus_queue.cc:237] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } }
I20260927 07:08:56.114564 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c830472b9c7f4fdc897f345209a1a3b2. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.114636 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: This master's current role is: LEADER
W20260927 07:08:56.111528 11924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:08:56.119540 11932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:56.119750 11932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:56.134382 11852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [term 1 FOLLOWER]: Refusing update from remote peer c830472b9c7f4fdc897f345209a1a3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:56.134774 11777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [term 1 FOLLOWER]: Refusing update from remote peer c830472b9c7f4fdc897f345209a1a3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:56.134764 11930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:56.135224 11913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:56.136278 11899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c830472b9c7f4fdc897f345209a1a3b2. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.136384 11899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:56.136579 11885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c830472b9c7f4fdc897f345209a1a3b2. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.136662 11885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:56.141448 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.119620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.21.2.124:45981)
0927 07:08:56.119674 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.119677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.123748 (+ 4071us) server_negotiation.cc:407] Connection header received
0927 07:08:56.123856 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.123858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.123869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.123900 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.127466 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.128186 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.129622 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.129921 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.129962 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.133340 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.133341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.133341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.133349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.139954 (+ 6605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.140385 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.140386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.140386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.140443 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.140628 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.140628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.140629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.140783 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.140891 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.141272 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.141348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:08:56.143666 11930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.143769 11930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:56.143851 11936 mvcc.cc:205] Tried to move back new op lower bound from 7333859066398605312 to 7333859066324377600. Current Snapshot: MvccSnapshot[applied={T|T < 7333859066398605312}]
I20260927 07:08:56.144413 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.119890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.21.2.125:42841)
0927 07:08:56.119972 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.119974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.123644 (+ 3670us) server_negotiation.cc:407] Connection header received
0927 07:08:56.125053 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.125054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.125069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.125108 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.125993 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.126758 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.130041 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.130338 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.130390 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.133652 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.133653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.133653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.133661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.138017 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.142439 (+ 4422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.142440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.142440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.142503 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.142997 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.142998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.142999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.143147 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.143232 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.144156 (+ 924us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.144302 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:08:56.144491 11885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.144567 11885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:56.144552 11939 mvcc.cc:205] Tried to move back new op lower bound from 7333859066398605312 to 7333859066324377600. Current Snapshot: MvccSnapshot[applied={T|T < 7333859066398605312}]
I20260927 07:08:56.144858 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.144941 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:56.145252 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:56.120104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.21.2.126:39939)
0927 07:08:56.120138 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.120140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.126153 (+ 6013us) server_negotiation.cc:407] Connection header received
0927 07:08:56.126253 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.126254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.126265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.126294 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.127006 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.127768 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.128788 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:56.129095 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:56.129137 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:56.132813 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:56.132814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:56.132814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:56.132821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:56.139192 (+ 6371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.142724 (+ 3532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.142726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.142726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.142788 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:56.143460 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:56.143461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:56.143461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:56.143628 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:56.143712 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:56.145087 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0927 07:08:56.145167 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:08:56.145249 11899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c830472b9c7f4fdc897f345209a1a3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c830472b9c7f4fdc897f345209a1a3b2" member_type: VOTER last_known_addr { host: "127.21.2.126" port: 39939 } } peers { permanent_uuid: "4151140fb752426ebf761e7fcda11dd6" member_type: VOTER last_known_addr { host: "127.21.2.125" port: 42841 } } peers { permanent_uuid: "6d9838bfb87b4ff081c831f9b85955eb" member_type: VOTER last_known_addr { host: "127.21.2.124" port: 45981 } } }
I20260927 07:08:56.145332 11899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:08:56.145972 11937 mvcc.cc:205] Tried to move back new op lower bound from 7333859066398605312 to 7333859066324377600. Current Snapshot: MvccSnapshot[applied={T|T < 7333859066398605312}]
I20260927 07:08:56.148193 11932 catalog_manager.cc:1347] Generated new cluster ID: fea9f4c6a5974433903ca6c7d87ef7a2
I20260927 07:08:56.148241 11932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:56.176815 11932 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:56.177614 11932 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:56.200359 11932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c830472b9c7f4fdc897f345209a1a3b2: Generated new TSK 0
I20260927 07:08:56.200699 11932 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:08:56.228394 11731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.226792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.21.2.126:39939)
0927 07:08:56.226839 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.226842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.227577 (+ 735us) server_negotiation.cc:407] Connection header received
0927 07:08:56.227757 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.227758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.227772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.227819 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.228321 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50266: BlockingRecv error: recv got EOF from 127.0.0.1:50266 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 07:08:56.229427 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.227222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.21.2.125:42841)
0927 07:08:56.227262 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.227263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.228896 (+ 1633us) server_negotiation.cc:407] Connection header received
0927 07:08:56.228996 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.228997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.229008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.229053 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.229368 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44708: BlockingRecv error: recv got EOF from 127.0.0.1:44708 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:08:56.230624 11880 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.229935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.21.2.124:45981)
0927 07:08:56.229983 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.229985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.230121 (+ 136us) server_negotiation.cc:407] Connection header received
0927 07:08:56.230215 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.230216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.230227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.230258 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.230562 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58812: BlockingRecv error: recv got EOF from 127.0.0.1:58812 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 07:08:56.233183 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.232431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.21.2.125:42841)
0927 07:08:56.232468 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.232470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.232671 (+ 201us) server_negotiation.cc:407] Connection header received
0927 07:08:56.232774 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.232775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.232790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.232819 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.233138 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44720: BlockingRecv error: recv got EOF from 127.0.0.1:44720 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 07:08:56.236368 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.235576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.21.2.125:42841)
0927 07:08:56.235610 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.235612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.235876 (+ 264us) server_negotiation.cc:407] Connection header received
0927 07:08:56.235967 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.235968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.235977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.236004 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.236325 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44734: BlockingRecv error: recv got EOF from 127.0.0.1:44734 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:08:56.240401 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.239560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.21.2.125:42841)
0927 07:08:56.239600 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.239602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.239892 (+ 290us) server_negotiation.cc:407] Connection header received
0927 07:08:56.240001 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.240002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.240011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.240052 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.240355 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44742: BlockingRecv error: recv got EOF from 127.0.0.1:44742 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 07:08:56.246369 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.245580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.21.2.125:42841)
0927 07:08:56.245614 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.245616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.245862 (+ 246us) server_negotiation.cc:407] Connection header received
0927 07:08:56.245955 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.245956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.245966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.245994 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.246326 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44756: BlockingRecv error: recv got EOF from 127.0.0.1:44756 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:08:56.256654 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.255728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.21.2.125:42841)
0927 07:08:56.255773 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.255776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.256072 (+ 296us) server_negotiation.cc:407] Connection header received
0927 07:08:56.256181 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.256183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.256194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.256225 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.256597 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44772: BlockingRecv error: recv got EOF from 127.0.0.1:44772 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:08:56.275069 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.274298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.21.2.125:42841)
0927 07:08:56.274349 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.274351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.274478 (+ 127us) server_negotiation.cc:407] Connection header received
0927 07:08:56.274519 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.274520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.274534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.274618 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.275007 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 07:08:56.309001 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.308257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.21.2.125:42841)
0927 07:08:56.308328 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.308330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.308450 (+ 120us) server_negotiation.cc:407] Connection header received
0927 07:08:56.308504 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.308505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.308515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.308567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.308909 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44790: BlockingRecv error: recv got EOF from 127.0.0.1:44790 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 07:08:56.374967 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.374191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.21.2.125:42841)
0927 07:08:56.374276 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.374278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.374378 (+ 100us) server_negotiation.cc:407] Connection header received
0927 07:08:56.374475 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.374476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.374489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.374561 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.374900 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44802: BlockingRecv error: recv got EOF from 127.0.0.1:44802 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 07:08:56.504889 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.504219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.21.2.125:42841)
0927 07:08:56.504294 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.504296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.504345 (+ 49us) server_negotiation.cc:407] Connection header received
0927 07:08:56.504405 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.504406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.504424 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.504480 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.504833 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44810: BlockingRecv error: recv got EOF from 127.0.0.1:44810 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 07:08:56.762774 11808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:08:56.762110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.21.2.125:42841)
0927 07:08:56.762158 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:56.762160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:56.762247 (+ 87us) server_negotiation.cc:407] Connection header received
0927 07:08:56.762309 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:56.762309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:56.762320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:56.762375 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:56.762731 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44816: BlockingRecv error: recv got EOF from 127.0.0.1:44816 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:08:57.063828 11896 catalog_manager.cc:1259] Loaded cluster ID: fea9f4c6a5974433903ca6c7d87ef7a2
I20260927 07:08:57.063927 11896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: loading cluster ID for follower catalog manager: success
I20260927 07:08:57.064733 11896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: acquiring CA information for follower catalog manager: success
I20260927 07:08:57.065047 11896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4151140fb752426ebf761e7fcda11dd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:08:57.080499 11910 catalog_manager.cc:1259] Loaded cluster ID: fea9f4c6a5974433903ca6c7d87ef7a2
I20260927 07:08:57.080569 11910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: loading cluster ID for follower catalog manager: success
I20260927 07:08:57.081307 11910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: acquiring CA information for follower catalog manager: success
I20260927 07:08:57.081542 11910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d9838bfb87b4ff081c831f9b85955eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:08:57.278362 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:57.276010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.21.2.125:42841)
0927 07:08:57.276100 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:57.276102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:57.276115 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:08:57.276174 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:57.276175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:57.276224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:57.276305 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:57.276442 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.276933 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.277714 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.277901 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.277962 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:57.278091 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:57.278196 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:57.278252 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:57.278282 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 07:08:57.278298 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260927 07:08:57.282297 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:57.280230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.21.2.124:45981)
0927 07:08:57.280328 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:57.280329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:57.280337 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:08:57.280370 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:57.280371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:57.280403 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:57.280470 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:57.280555 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.281010 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.281648 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.281829 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.281876 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:57.281998 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:57.282081 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:57.282133 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:57.282169 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 07:08:57.282185 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":52,"threads_started":1}
I20260927 07:08:57.286657 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:57.283794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.21.2.126:39939)
0927 07:08:57.283899 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:57.283901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:57.283931 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:08:57.284016 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:57.284017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:57.284068 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:57.284176 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:57.284278 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.284808 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.285791 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.286077 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.286129 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:57.286318 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:57.286417 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:57.286487 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:57.286539 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 07:08:57.286562 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260927 07:08:57.286717 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:57.283806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.21.2.125:42841)
0927 07:08:57.283853 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:57.283855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:57.284076 (+ 221us) server_negotiation.cc:407] Connection header received
0927 07:08:57.284180 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:57.284180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:57.284191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:57.284244 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:57.284427 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.285119 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.285975 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.286232 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.286292 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:57.286436 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:57.286465 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:57.286522 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:57.286623 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:08:57.286647 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:08:57.287108 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:57.283978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.21.2.124:45981)
0927 07:08:57.284013 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:57.284015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:57.284059 (+ 44us) server_negotiation.cc:407] Connection header received
0927 07:08:57.284132 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:57.284133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:57.284142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:57.284187 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:57.284325 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.285042 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.286331 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:57.286602 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:57.286658 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:57.286829 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:57.286861 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:57.286890 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:57.287017 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 07:08:57.287041 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:08:57.293355 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11660
I20260927 07:08:57.298583 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11735
I20260927 07:08:57.303633 21513 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHv0k7x/build/release/bin/kudu with pid 11810
2026-09-27T07:08:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2815 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5605 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (295606 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS